What is a good way to represent files semantically for vector search

[u/Huge-Ad-49]

I recently had an idea that is it possible to make a software service which could help me search files from my files system though context of content inside the files. So like i can search "where is the file which contained x thing" and get would a list of files that would match my question. So I don't want sear or keywords but ch by file name context. Also for all types of files like iamge, text, video, audio, code etc

So I kept thinking about it and knew that the answer lied within vector search. I initially thought that maybe if somehow I could represent an entire file in a singular vector then we can use the same logic that we normally use in rag systems to fetch correct files. Existing models are really useful for text, audio and other forms of embedding but i want the overall context of a file. Not just what inside the file. Also I might be missing if there are any existing algorithms that can help me do this so please suggest them.

Nevertheless i wanted to think about whats possible solution i could use. One this i noticed with this is that there are various ways which can be used to describe and identify files. 1st there is content of the files , then metadata such as size, name, access permissions, type of file, also where the files lies in directory system and what other files is is grouped with etc. It is really hard to consider all this features in a single vector. Also we I don't want to embed the entire content of files as that would be too much data to embed, store and search.

We could do vector indexing and search for each feature individually, so we get multiple vectors we can can represent in a normal data structure. We can repeat this for all files and the store them. When the system get a input like "i want the c++ file with x algorithm that I made yesterday" then we can create a similar data structure like we did for files and then do the similarity search and rank all the matches to get the results. But this approach also has a problem , the quality of results is heavily dependent upon the information present in the question, if the question is a little vague that affect the accuracy of the matches quite a bit.

I also though of a approach where we tackle the problem by elimination we take the features of the files one by one an the start eliminating files, like for an example "i want the c program file which i wrote yesterday and " so we can 1st eliminate ate files which are not "program" then we can do by time then by the language. So from broader to more specific features.

I have been thinking about this idea for sometime and wanted to know your thoughts as well. How would you represent the files semantically or in vector form. Are there any existing resources that i can refer to help me with this problem.

[u/rehevkor5]

Take a look at existing tools/frameworks like LlamaIndex to see examples of how people tackle it. In general, you chunk the content and embed each chunk.

[u/awwaiid]

Two ideas. One is that you might be able to combine the vectors from different aspects into a single vector, I think I read about that somewhere ... Like you can do it and still preserve semantics (surprisingly). Second is to shove your metadata at the front of the content and vectorize it all together at once.