Python — public feed snapshot · snapshot for July 2025

31 July 2025

Thank you, but how fast is vectorization? Is it done directly in the database? I just tried the ClickHouse database and found it very troublesome.

09:31

Артём🇷🇺

1 1Thank you, but how fast is vectorization? Is it done directly in the database? I just tried the ClickHouse database and found it very troublesome.

I didn't test it, so I can't say for sure. Check the info online :)

09:31

1 1

Артём🇷🇺I didn't test it, so I can't say for sure. Check the info online :)

OK, I'll check and try it.

09:34

I'm currently using hash duplication detection, but I think it's too slow. I've heard that deduplication can be implemented within the database. I'd like to know which database can do this.

09:41

Does anyone know how to detect duplication in large amounts of data?

09:42

Jürgen Romins

1 1I'm currently using hash duplication detection, but I think it's too slow. I've heard that deduplication can be implemented within the database. I'd like to know which database can do this.

any

09:42

1 1

Jürgen Rominsany

I am implementing a quick detection of similarity, which is currently equivalent to checking for plagiarism in papers, but I need to have fewer words. Do you mean that all databases support this? I would like to ask for your advice.

09:44

I just tried to calculate the hash of the text that needs to be checked for duplicates in the database and then pull it into memory, and then use my python program to check it, but I found that this was too slow.

09:48

Jürgen Romins

1 1I am implementing a quick detection of similarity, which is currently equivalent to checking for plagiarism in papers, but I need to have fewer words. Do you mean that all databases support this? I would like to ask for your advice.

if need to have uniq yes all db supporting it

09:53

1 1

Jürgen Rominsif need to have uniq yes all db supporting it

No, brother, my data is currently in MySQL. In MySQL, I have a field called is_repeat. When a duplicate is detected, this field needs to be updated. What I am thinking now is that as long as the text similarity is greater than 50%, the two will be considered duplicates. Brother, do you have any good ideas?

10:02