UnQLite Users Forum

Document store performance? Indexes?

append delete Alex


After reading UnQLite's description and tutorials, I really wonder about the performance of performing "SQLite-like queries" using the document store. I understand that one can perform queries using db_fetch_all() and a filter callback, but what about the performance of this compared to an indexes SQLite table? Is UnQlite building some indexes transparently to avoid parsing the whole collection? Or is it possible to explicitly ask it to create indexes and speed up queries?

Reply RSS


append delete #1. chm

Yes, retrieved records from a given collection a parsed only once (FastJSON) and cached in-memory for fast retrieval. The cache is a O(1) hashtable so performance shouldn't be a problem.

append delete #2. Alex

Thanks for the reply. So does it mean the first retrieval might be slower than others? I am thinking about replacing a SQLite query/storage application with UNQLite, for it would make queries building much easier, but am worried about performance (and would like to understand how it works altogether). Using indexes, SQLite queries are blazing fast even with millions of records, and I'd like to make sure I won't lose this.

append delete #3. devel

Yes, only the first retrieval is a little bit slower than the others since it involve a disk access (In case you are working with an on-disk DB) and a preprocessing phase (FastJSON decoding) while later fetches are in-memory operations only with no decoding overhead.

Honestly, I haven't benchmarked SQLite indexes with UnQLite doc store. But, keep in mind that UnQLite is shipped with a *disk* KV storage engine that support nearly O(1) record retrieval.
For a clear idea, I recommend you to start an experimental UnQLite portage and do some performance test against SQLite indexes.

append delete #4. Alex

Yep, I will do that and report the results here. UnQLite looks promising in any case. Thanks for the answers.

append delete #5. nibedit

My Observations about document storage in UnQlite :
In memory: Store faster, search slower ;But both slower than SQLite !
In File: Store slower, search faster
e.g:- For same data, In UnQLite:
In mem: store: 901.9 ms ,Search: 2586 ms
In Local disk: store: 15223.3 ms , search: 114.2 ms
In mem: Store 9507 ms , Search : 105 ms

@Alex: Please update ur observations too . Please correct me ,if any of my results are wrong.

append delete #6. devel

The performance of the UnQLite Perl binding shows that UnQLite outperforms SQLite in many phases especially under the KV Store layer . See this slide: http://www.slideshare.net/charsbar/unqlite

@nibedit I'm interested in your benchmarks. Could you make them available.

append delete #7. Kosta

@nibedit: can you give some more info about what you are storing and what you are querying? The raw numbers don't help much...

append delete #8. Mohamed Chit

Dear All,
The question is very important, unfortunately I do not see an answer describes fully the perfomance in terms of complexity.

May I rephrase the question:
when you perform queries using db_fetch_all () and a filter callback. Is the search always linear ?
if not, what does UnQLite to speed up the query ? the query filter is defined in my callback function, how speeding up the query is done by UnQLite.

append delete #9. devel

The document store layer is just a wrapper around the Key/Value store which is optimized (With the adequate storage engine) for on-disk as well on-mem operations.

append delete #10. Mohamed Chit

Thank you for the reply,

The Key/Value store is optimised for the key, when I write a script in j9x and I write a custom function to filter JSON objects in a collection, I think the search is still linear.

Could you please clarify this point:
When you perform queries using db_fetch_all () and a user defined filter callback. How this can be used in the Key/Value store ?

Could you please confirm, is the search always LINEAR when using j9x user defined filter callback function ?



(Leave this as-is, it’s a trap!)

There is no need to “register”, just enter the same name + password of your choice every time.

Pro tip: Use markup to add links, quotes and more.

Your friendly neighbourhood moderators: chm_at_symisc, devel_at_symisc