### What is op-storage

A simple interface for interacting with MongoDB


### What problem does it solve

Modern DBs are powerful, but interacting with them rapidly gets complex, which often leads to errors and increased development time. For the vast majority of use cases, you won't use most of the features. If we omit these advanced features and leave in just the "essentials", we can simplify how we interact with our data without sacrificing performance.

Furthermore, in an effort to be cross-platform, many python DB adapters do not do things in a "pythonic" way (e.g. there is often a need to specify explicit schemas / typing). Most ORMs designed to work with SQL model data as classes, and the abstraction often breaks - you'll most likely have to write some SQL at some point, even if it's just creating an index and not anything from your code. Document stores like MongoDB are better, but often have limitiations on the types of data that can be stored and used in indexes natively (e.g. no 'date' type, reserved special characters in field names).

I want to hide the implementation details completely for the sake of simplicity and to avoid esoteric concepts. The op-storage interface could be implemented over a variety of different backing DBs.


### Essential features

- Document model with plain old dictionaries, no object classes

- IDs are not properties of objects, they are "pointers" to objects

- CRUD operations

- Indexed filter operations on any field(s) of the following types:
  bool, str, int, float, bytes, datetime, date, UUID, Decimal

- Create with unique constraints

- Optimistic locking on updates

- Get and delete objects in bulk by ID

- No type restrictions, including on document keys

- No naming restrictions for object namespaces and fields

- No SQL, no esoteric query structure, write queries using native python operators

- No need for upfront schema declaration

- Simplify index creation


### Quick-start

``` python
import op_storage

# The arguments to the build function depend on your DB implementation. For MongoDB, you should have a connection
# string, and your call could looks as follows:
# db = factory.build('mongo_db', { 'connection_string': 'CONENCTION_STRING', 'database': 'my-db-name' })
# See the 'Development' section at the end of this document for info on how to bring up a local MongoDB instance to
# experiment with:
db = factory.build(...)

user_id = db.create('user', { 'name': 'Alex', 'age': 35, 'height': { 'inches': 73 } })

user = db.get('user', user_id)
print('User:', user)

db.update('user', user_id, { 'name': 'Alex', 'age': 36, 'height': { 'inches': 73 } })

users = db.list('user', 30 < op_storage.Field('age') < 40)
for user_id, user in users:
  print('User:', user)

db.delete('user', user_id)
```


### Write query conditions using python syntax

ORMs and other database adapters have implemented query functionality in different ways. Some outright allow you to embed pseudo-sql as a string. Few stick to native python constructs. For example, given a raw SQL query as follows (SQL doesn't support nested document properties, so I'm assuming a slightly different structure and ignoring the complexities of adapting our code to that):

``` sql
SELECT * FROM user WHERE age >= 30 AND age < 40 AND height_inches = 73;
```

In MongoDB the same query is written as follows - note that we do need to know about a few special keywords:

``` python
user.find({ 'age': { '$ge': 30, '$lt': 40 }, 'height': { 'inches': { '$eq': 73 } } })
```

Using op-storage interface, the same query is written as:

``` python
db.list('user', 30 <= Field('age') < 40, Field('height', 'inches') == 73)
```


### Object IDs 

SQL DBs do not necessarily enforce that a table contains an indexed ID field. Traditionally, an ID field is explicitly created using an auto-incrementing integer, which is generally considered bad practice as it can reveal the size of your data set. The best practice is to use something like a UUID as the primary key. No-SQL databases such as MongoDB typically do a good job of automatically attaching an ID to every object, although sometimes the implementation can get murky. For example, MongoDB explicitly injects an `_id` property into your object, which could lead to conflicts.

The op-storage interface automatically generates UUIDs for every object and stores them separately without ever attaching additional ghost properties to objects. This even allows you to set an object to null:

``` python
user_id = db.create('user', { 'name': 'Alex', 'age': 35 })
db.get('user', user_id) # { 'name': 'Alex', 'age': 35 }
db.list('user') # [ (<user_id>, { 'name': 'Alex', 'age': 35 }) ]
```


### On indexing

Making sure you have the right indexes in place to avoid scanning all documents on every query is important and often difficult. op-storage provides methods to see and update indexes directly from code. Configuration options for these indexes are highly limited, but are sufficient and efficient in the vast majority of real-world use cases.

For example, say we wanted to query users by both name and age:

``` python
db.create_index('user', Field('name'), Field('age'))
```

Or if we added an 'email' property to users and wanted to ensure it was unique:

``` python
db.create_index('user', Field('email'))
```


### Shortcomings

- The op-storage interface does not necessarily minimize storage space. A tightly controlled SQL table with very specific types will take up less space. It also uses more space than pure MongoDB to allow for any data type and to add support for dates, decimals, and others.

- No numeric indexing into stored lists when filtering. The key-paths passed to Field objects can only be dictionary keys (although they do not necessarily need to be strings).

- No bulk insert / bulk update. This interface was not designed for porting around big data. I have had cases where I need to backfill properties selectively in hundreds of thousands of objects. I used the method of iterating over the objects and updating as needed one by one (my loop was multi-threaded). Easy to write, but slow to execute. MongoDB itself does provide bulk methods if updating one-by-one is untenable (however, see next point).

- In rare cases, it might be necessary to exploit some of the features of the underlying DB implementation, for example through web interfaces or CLI tools provided by those DB services. Data stored by op-storage may be difficult to work with in those cases.

For example, given the following call to create an object:

``` python
db.create(
  'user',
  {
    'name': 'Alex',
    'age': 35,
    'net_worth': Decimal('16.00'),
    'spouse_user_id': UUID('67521ac5-852d-4a5b-ab47-594e2c581124'),
    'child_user_ids': [ UUID('8df3054c-618a-4d15-808a-fcb2c6cf500f') ],
    'last_travel_datetime': datetime.datetime(2020, 1, 1, 0, 0, 0, 0, tzinfo=datetime.timezone.utc),
    'last_travel_date': datetime.date(2020, 1, 1),
    'exception': Exception('Message'),
    'nested': { 'a': 1, 'b': 2 },
    '$_?': 'special characters in field name'
  }
)
```

The underlying document in MongoDB actually looks something like this:
``` python
{
  '_id': UUID('f5ca9e16-1b61-4724-9873-b774f77caaec'),
  '+': {
    'name': 'Alex',
    'age': 35,
    'net_worth': Decimal128('16.00'),
    'spouse_user_id': UUID('67521ac5-852d-4a5b-ab47-594e2c581124'),
    'child_user_ids': {
      '?': b'\x80\x04\x953\x00\x00\x00\x00\x00\x00\x00]\x94\x8c\x04uuid\x94\x8c\x04UUID\x94\x93\x94)\x81\x94}\x94\x8c\x03int\x94\x8a\x10]=\xc9\x8f\\\xef\xa4\x81\xdfAF\x1f\xaf\xfd\xc6Ksba.'
    },
    'last_travel_datetime': datetime.datetime(2020, 1, 1, 0, 0, 0, 0, tzinfo=datetime.timezone.utc),
    'last_travel_date': datetime.datetime(2020, 1, 1, 0, 0),
    'exception': {
      '?': b'\x80\x04\x95(\x00\x00\x00\x00\x00\x00\x00\x8c\x08builtins\x94\x8c\tException\x94\x93\x94\x8c\x07Message\x94\x85\x94R\x94.'
    },
    'nested': { 'a': 1, 'b': 2 },
    '?D_??': 'special characters in field name'
  },
  '-': b'\x80\x04\x953\x00\x00\x00\x00\x00\x00\x00\x8c\tnet_worth\x94\x85\x94\x8c\x01D\x94\x86\x94\x8c\x11last_travel_date\x94\x85\x94\x8c\x01d\x94\x86\x94\x86\x94.'
}
```

---


# Development


Run the tests - note that the tests on having a local instance of MongoDB running:

``` bash
docker pull mongo:latest
docker run -p 27017:27017 mongo:latest
```

``` bash
python3 -m unittest
```

Build and install the package locally:

``` bash
pip3 uninstall op-storage -y
pip3 install .
```

Build the package for distribution:

``` bash
python3 setup.py clean --all && python3 setup.py build && python3 setup.py sdist
```
