Php in memory storage

In memory storage: PHP shared memory vs Redis vs MYSQL

Continuing on the theme of rethinking the data core of emoncms, as previously mentioned for short term storage, storage to disk may not be necessary, instead we can store data in memory using an in-memory database. Here are some tests and benchmarks for in memory storage:

To start with I created baseline test using MYSQL updating a feed’s last time and value in the feeds table row for that feed. This took around: 4800 – 5100 ms to update a row 10000 times.

We would imagine Redis doing a lot better as its in-memory, it isn’t writing to disk each time which is much slower than memory access. Redis did indeed perform faster completing the same number of updates to a key-value pair in 1900 – 2350ms. I’m a little surprised thought that it was only 2.3x as fast and not much faster, but then there is a lot going on Redis has its own server which needs to be accessed from the PHP client this is going to slow things down a bit, I tested both the phpredis client and Predis. Phpredis was between 500-1000 ms faster than the Predis client and is written in c.

How fast can in-memory storage be? A general program variable is also a form of in-memory storage, a quick test suggests that it takes 21ms to write to a program variable 10000 times, much better than 2.3x faster that’s 230x faster! The problem with in program variables is that if they are written to in one script say an instance of input/post they cannot be accessed by another instance serving feed/list, we need some form of storage that can be accessed across different instances of scripts.

The difference between 21ms and 1900-2350ms for redis is intriguingly large and so I thought I would search for other ways of storing data in-memory that would allow access between different application scripts and instances.

I came across the PHP shared memory functions which are similar to the flat file access but for memory, the results of a simple test are encouraging showing a write time of 48ms for 10000 updates. So from a performance perspective using php shared memory looks like a better way of doing things.

The issue though is implementation, mysql made it really easy to search for the feed rows that you wanted (either by selecting by feed id or by feeds that belong to a user or feeds that are public), I’m a little unsure about how best to implement the similar functionality in redis but it looks like it may be possible by just storing each feed meta data roughly like this: feed_1: <"time":1300,"value":20>.

Shared memory though looks like it could be quite a bit more complicated to implement, but then it does appear to be much faster. Maybe the 2.3x speed improvement over mysql offered by redis is fast enough? and its probably much faster in high-concurrency situations. I think more testing and attempts at writing full implementations using each approach is needed before a definitive answer can be reached.

Recent Posts

Источник

Saved searches

Use saved searches to filter your results more quickly

You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session.

A simple in-memory data storage for PHP

License

scheb/in-memory-data-storage

This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.

Name already in use

A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch?

Sign In Required

Please sign in to use Codespaces.

Launching GitHub Desktop

If nothing happens, download GitHub Desktop and try again.

Launching GitHub Desktop

If nothing happens, download GitHub Desktop and try again.

Launching Xcode

If nothing happens, download Xcode and try again.

Launching Visual Studio Code

Your codespace will open once ready.

There was a problem preparing your codespace, please try again.

Latest commit

Git stats

Files

Failed to load latest commit information.

README.md

A fast in-memory data storage in plain PHP. It can be used as a test double for a database, in order to to decouple the test cases from the database and speeding them up.

• CRUD operations
• Convenience methods for data selection and manipulation
• Named items

composer require scheb/in-memory-data-storage

You can find an executable example in the doc folder.

 use Scheb\InMemoryDataStorage\DataRepositoryBuilder; use Scheb\InMemoryDataStorage\DataStorage\ArrayDataStorage; use function \Scheb\InMemoryDataStorage\Repository\compare; $foo = 'I am foo'; $bar = 'I am bar'; $repositoryBuilder = new DataRepositoryBuilder(); $repository = $repositoryBuilder // ->setDataStorage(new ArrayDataStorage()) ->build(); // Simple CRUD $repository->addItem($foo); $repository->containsItem($foo); // returns true $repository->getAllItems(); // returns [$foo] $repository->removeItem($foo); // Named items $repository->setNamedItem('foo', $foo); $repository->namedItemExists('foo'); // returns true $repository->getNamedItem('foo'); // returns $foo $repository->replaceNamedItem('foo', $bar); $repository->getNamedItem('foo'); // returns $bar $repository->removeNamedItem('foo'); // Advanced get $repository->getAllItemsByCriteria(['property' => 'value']); // $repository->getOneItemByCriteria(. ); // The same, but only one item is retrieved // Advanced update $repository->updateAllItemsByCriteria( ['property' => 'value'], // Match criteria ['property' => 'newValue', 'otherProperty' => 42] // Property updates ); // $repository->updateOneByCriteria(. ); // The same, but only one item is updated // Advanced remove $repository->removeAllItemsByCriteria(['property' => 'value']); // $repository->removeOneItemByCriteria(. ); // The same, but only one item is removed // Comparision functions in matching criteria $repository->getAllItemsByCriteria(['property' => compare()->notNull()]); $repository->getAllItemsByCriteria(['property' => compare()->lessThan(3)]); $repository->getAllItemsByCriteria(['property' => compare()->between(1, 3)]); $repository->getAllItemsByCriteria(['property' => compare()->isInArray([1, 2, 3])]); $repository->getAllItemsByCriteria(['property' => compare()->arrayContains('arrayElement')]); $repository->getAllItemsByCriteria(['property' => compare()->dateGreaterThan(new \DateTime('2018-01-01'))]); // . and many more, see Scheb\InMemoryDataStorage\Comparison

Single-item (read, update, remove) operations will handle gracefully, when there is no matching item in the data store. Reads will return null , update/remove won’t change anything. If you want such cases to raise an exception instead, you can configure that behavior with the builder for each type of operation.

$repository = $repositoryBuilder ->strictGet() ->strictUpdate() ->strictRemove() ->build();

The library comes with a simple array-based storage, but you exchange the data storage engine with whatever you like, by implementing Scheb\InMemoryDataStorage\DataStorage\DataStorageInterface . Then, pass an instance to the builder:

$repository = $repositoryBuilder ->setDataStorage(new MyCustomDataStorage()) ->build();

Values are checked for equality with PHP’s === operation. If you want it to use the less-strict == operator, you can change the behavior with the builder.

$repository = $repositoryBuilder ->useStrictTypeComparison(false) ->build();

The library integrates scheb/comparator to check values for equality. If you need custom comparison rules, implement Scheb\Comparator\ValueComparisonStrategyInterface and pass your comparision strategy to the builder.

$repository = $repositoryBuilder ->addComparisonStrategy(new MyCustomValueComparisonStrategy()) ->build();

You can add as many comparision strategies as you need. They’ll be checked in the order they’re added and will take preference over the default comparision strategy.

If you want to implement the comparision logic completely on your own, implement Scheb\InMemoryDataStorage\Matching\ValueMatcherInterface and pass an instance to the builder:

$repository = $repositoryBuilder ->setValueMatcher(new MyCustomValueMatcher()) ->build();

Properties are read and written with the following access strategies:

If none of these works on a value object, it’s either using null (in case of read) or fails with an exception (in case of write).

If you want to implement the property access logic completely on your own, implement Scheb\PropertyAccess\PropertyAccessInterface and pass an instance to the builder:

$repository = $repositoryBuilder ->setPropertyAccess(new MyCustomPropertyAccess()) ->build();

You’re welcome to contribute to this library by creating a pull requests or feature request in the issues section. For pull requests, please follow these guidelines:

• Symfony code style
• PHP7.1 type hints for everything (including: return types, void , nullable types)
• Please add/update test cases
• Test methods should be named [method]_[scenario]_[expected result]

To run the test suite install the dependencies with composer install and then execute bin/phpunit .

This bundle is available under the MIT license.

Источник

In-memory data cache architecture

Here is the design I came up with, which is tested and fully functional.

There are too many Exceptions for my taste, but somehow I need to enforce things that «should not happen». Another thought was to log errors silently and then ignore them. However, that means my code using this API will have to account for those errors, increasing the complexity.

The basic methods are create , readOne , read , and delete . CRUD usually includes update but I find it useless as it can be accessed by combining ‘read’, ‘delete’, ‘create’ and requires additional decision how to handle concurrency.

I’m curious to get feedback and thankful for any critiques or remarks regarding what can be improved.

/** Local In-Memory Cache: Tables referred by their string names * Storing hashes: * all hashes must have 'id' as key * all hashes must have keys other than 'id' * and hashes in the same table must have the same key structure (enforced) * Basic CRUD interface + 'readOne' for the first entry + 'realDelete' * optimized to save memory: arrays packed into |-separated strings and gz(de/in)flated */ class LocStore < // ass. array 'table' =>string of keys private $_keys = array(); // assoc. array of strings indexed by table and id private $_values = array(); /** create entry, enforce the same keys and new id * @param string $table * @param ass. array $hash, must have 'id' not null */ public function create ($table, array $hash) < $id = $hash['id']; if (! $id) throw new Exception('Id is null'); unset($hash['id']); if (! $hash) throw new Exception('Empty hash beside id'); // sort by keys alphabetically ksort($hash); $keyString = $this->_toString(array_keys($hash)); if ( empty($this->_keys[$table]) ) < $this->_keys[$table] = $keyString; > elseif ($this->_keys[$table] != $keyString) < throw new Exception('Array keys mismatch'); >if ( isset($this->_values[$table]) && ! empty($this->_values[$table][$id]) ) < throw new Exception("Id '$id' already exists"); >$this->_values[$table][$id] = $this->_toString(array_values($hash)); // for chaining return $this; > // read one entry, empty array if nothing is there public function readOne ($table) < if ( empty($this->_values[$table]) ) return array(); reset($this->_values[$table]); $id = key($this->_values[$table]); return $this->read($table, $id); > // read by id, empty array if not found public function read ($table, $id) < if ( empty($this->_values[$table]) || empty($this->_values[$table][$id]) ) return array(); $keys = $this->_toArray($this->_keys[$table]); $values = $this->_toArray($this->_values[$table][$id]); $result = array_combine($keys, $values); $result['id'] = $id; return $result; > // delete by id, exception if not found public function delete ($table, $id) < if ( empty($this->_values[$table]) || empty($this->_values[$table][$id]) ) < throw new Exception('Deleting non-existant entry'); >unset($this->_values[$table][$id]); return $this; > public function readDelete($table, $id) < $hash = $this->read($table, $id); if ($hash) $this->delete($table, $id); return $hash; > private function _toString (array $array) < $json = json_encode($array); return gzdeflate($json); >private function _toArray ($string) < $string = gzinflate($string); return json_decode($string); >> 

Источник