Overview

• acls are based on limiting access to certain plugins
• in particular: 'editor' and 'external' plugin types
• a user has access to any, no or a subset of plugins
• the admin flag on a user account disables all ACL checks for that account
• only admin users are allowed to:
  • add/remove ACLs
  • add/remove Accounts
  • reset passwords
• if there is no permission for a plugin, it should not show up in bars
• there must be a error message if it is still called (e.g. by passing the right GET data)
• by default a new user account is not restricted (this should be an option at the time a account is created)
• the admin flag is not set for new accounts

Technical Specification

• ACLs should be verified in stockmaniac::run_plugin(), e.g by implementing stockmaniac::check_acl()
• additional check in stockmaniac::create_bar() to remove restricted ones from all bars
• the Settings plugin must repect ACLs (perhaps show restricted ones, but do not allow to change them --> that requires a ACL check during the POST data processing!)
• load ACLs during login phase and store them in stockmaniac::$USER[acl]
• no ACL record found means access is denied
• the 'Access' column is defined as 0/1 switch where 1=granted and 0=prohibited
• see README.schema (FIXME: link) for details on the table layout

Design

documents are stored inside the database as a Binary Large OBject (BLOB)

it is supposed to be robust

• a maximum upload size is calculated from certain system parameters
• the calculated size is very defensive in order to avoid memory problems in the first place
• many checks are performed to detect even the slightest anomalies in all stages of the upload process

document storage is spread over multiple tables to increase performance

• table 1 holds meta informations such as mimetype, filename, ..., etc.
• table 2 holds the content of all files (and nothing else)
• additional tables are used to assign documents to various other items. That might be Positions, Orders, ..., etc.

a document does *always* belong to a user

• it might be assigned to various items such as Portfolio, Position, Order, etc...
• assignments can be extended to other items if necessary

Known Issues

database column size is hardcoded in order to avoid that the DBMS truncates the file content silently. Look in the code for details.

the upload code is quite inefficient in regards to memory consumption. Ie during development I tried to upload a 20 MByte file. To achieve that it was necessary to set:

     (php.ini) memory_limit       = 100M
     (my.cnf)  max_allowed_packet = 45000000

     (php.ini) post_max_size       = 25M
     (php.ini) upload_max_filesize = 25M

• http://php.net/mysql_real_escape_string() is used to make file data SQL safe. It seems to be very ineffective when operating on binary strings. Perhaps it might be even better to http://php.net/base64_encode() file data first?
• it looks like file data is copied too often in memory (use references?)

base64 encoding might make it more robust?

Speed Things Up (to be implemented later)

Apache (meaning any HTTP server in this context) can serve files from the filesystem much faster than MySQL (meaning any RDBMS in this context) can read BLOBs out of some table and serve that to Apache, which will then serve it to the browser...

Despite the performance there are quite some advantages in having the documents inside the RDBMS. It ranges from easier backup to less logic in the application (no need to write a 'storage layer' to distribute the files across a directory structure to avoid to many files per directory. No need to find a way to ensure consistency between file storage and database...).

For these (and some other) reasons I decided to store all documents inside the RDBMS. If performance becomes a issue one time, the plan is to implement a cache feature on top of the database storage.

The cache is actually a area on the file system containing all documents as regular files. This way Apache can serve files directly and thus downloads become much faster.

Data consistency wont be an issue because the cache is read-only in the sense that new documents are never written directly to the cache. They should go into the RDBMS first. Then get stamped with a unique ID, which will be the basis for building the cache.

further requirements:

• cache is optional so that it can be switched on or off at any time
• security: user A should not be able to fetch documents from user B just by knowing (or guessing) the document ID
• security: browsing the cache via HTTP should not be possible
• the cache will be rebuild from scratch if there is the slightest sign of something being messed up (meaning we wont try to fix the cache)

On-the-Fly Data Compression (to be implemented later)

• support multiple algorythms as available in PHP (gzip, zip, bzip2, ...)
• decompress on-the-fly before download
• optional compressed download (useful if on a slow line)
• administrators can configure system default compression (compression on/off, algorythm used)

Design

RSS feeds are integrated as a two-tier design. This came up because I want news lookups happen automatically in the background. Meaning:

• no intervention from the user side (manual updates are possible, but not required)
• no additional cron job (ie no standalone php script using MagpieRSS)

Tier 1: MagpieRSS library (PHP)

• lookup feeds from web sites and/or URLs
• create system wide NewsFeed records from gathered informations
• lookup related News for positions
• this tier is implemented in the ObjectManager and PositionNews plugins

Tier 2: XML::RSS library (Perl)

• lookup news items for all feeds in the system
• this tier is implemented in the ObjectUpdater plugin as part of the quote retrieval process
• able to run as independend background job

For that reason StockManiac will strip down news items to a few informations that are considered essential for doing what StockManiac does. Look at the NewsItem table design for details.

Identifying News Items

... really is a problem with RSS. Since there is no unique identifier for news items (versions 0.9x and 1.0 do not specify it, 2.0 does - but it is not required). For that reason StockManiac has to create a (more or less) unique string in order to identify a news item within a feed.

The NewsItem table provides a general 'Indentifier' field for this purpose. A identifier is a string that is either coming from the actual news source (ie a RSS guid or the atom:id field, ...) or it is generated somehow. The essential requirement is that the string is unique enough to identify a news item within its feed. Its purpose is to see whether we have fetched a new item already.

The Identifier is *not* used for anything other than that. News assignments, read/unread marks, etc... are implemented by utilizing the internal database IDs which are usually AUTO_INCREMENT values and therefore much more reliable.

Identifiers are supposed to be extensible in the future. Ie. we could use different hashing algorythms and/or store some entity from a external source in there. The 'IdentifierMethod' field is ment to indicate what exactly a identifier is. This way code can know how to deal with the identifier string.

Currently there is only 'md5' defined as identification method. Extensions should be added to the enum list.

RSS Protocol Versions

StockManiac *should* support everything that MagpieRSS and XML::RSS can handle. However, while implementing I noticed that 7 out of 10 feeds are not conform with any of the existing RSS versions.

Sometimes there are fields missing, at other occasions fields are not formatted correctly. Ie pubDate should be a RFC822 formated string. I have seen all kinds of date strings: date with/without time, as 12h, as 24h, with/without timezone, etc... some feeds do not even mention a version.

So the current implementation is my-best-guess based on the feeds I tested with. There is a lot room for improvements.

StockManiac is calculating a MD5 hash for each news item in order to figure whether the item has been fetched already (see above). It is using the following fields:

	link  + pubDate  + strlen(title)

• it does *not* use the 'title' field itself, only its length
• if a field is NULL ('', undef) it is silently ignored and the remaining fields make up the basis for the hash

Atom

Atom is currently not supported. It will be. I just havn't done it yet because all the feeds I am interested in are RSS formatted. It seems that atom is technically much nicer but is not used widely yet (?). Like many good things.

However, I'll probably implement it one time. You can increase the priority by requesting it or by submitting a patch - this will be even faster. :-)

The atom:id element should be stored in the Hash column, as it is per definition unique [enough to identify a news item]. I would suggest to add a new IdentifierMethod. Maybe 'atom:id' or just 'atom'.

Plugin Types

Each plugin must mention one out of several possible types in it's init.php file. Here is a list of currently implemented types and their meaning.

[integrated]

plugins of this type will be 'merged' into StockManiacs Main Page. They can be a Editor, a Day Selection Bar, some Display or anything else that is useful in this position

It is not necessary to create a new stockmaniac object within a integrated plugin since the one created in index.php is available due to the include mechanism. To access it use:

do NOT implement HTML Headers and Footers!

Integrated plugins are not added automatically. Since StockManiac can not know where you want to see it. So you must call it's run.php file from within index.php and add it's template to index.tpl at your favorite position

The SelectDay plugin (FIXME: link) is a good example for integrated plugins

[editor]

Plugins of the type 'editor' are integrated into StockManiacs Main Page in exactly the same way as 'integrated' (FIXME: link) plugins. Thus you can access the global StockManiac Object as well.

Editors are automatically added to the Editor Selection Bar (below each editor).

InfoDisplay (FIXME: link) is a good example for editor plugins

Plugins of this type are external pages that can not be merged into StockManiacs Main Page. Therefore they have to implement a complete HTML page with Header and Footer (of course the may use the provided header.tpl and footer.tpl templates)

External plugins can not access index.php's StockManiac object. So they _have_ to create a new object.

External plugins are automatically added to the Feature Selection Bar (on top of the Main Page). See ObjectManager (FIXME: link) as example for external plugins

External plugins have to implement a ACL check right at the beginning. That is as soon as the StockManiac object is available. The class can not do this right now because of the way things are designed/implemented.

ACL check sample code

[hidden]

Plugins of this type are detected by StockManiac just like any other plugins.

Hidden plugins are *never* automatically loaded and/or included anywhere. You can call them inside your code if you *know* they are there

Hidden plugins should never be offered to the user and might even be 'headless', that is they may do something without producing any visible output on the GUI.

Hidden plugins may or may not create their own object(s). Take the Migration plugin (FIXME: link) as example for hidden plugins

Naming Convention for Template (Smarty) Variables

To avoid interferences between plugins it is important that each plugin sets unique variable names for its template(s). Interferences may occur especially with integrated or editor plugins since they are all merged together in index.tpl. So please follow this simple convention while writing plugins.

A variable that is assigned to a template should be named:

	<plugin name>_<some meaningfull variable name>

SQL Queries inside Plugins

There are basically two ways to execute a query:

(1) call a pre-defined query function via a stockmaniac or dbc object. Please check dbi.class.php for available queries. If you can not find what you need proceed with number (2) below.

(2) run a custom query via the dbc::sql() method ( which is inherited by the stockmaniac class)

Custom queries should be avoided if possible to keep the total number of SQL queries in the appliacation low (as of writing this there are 200+ queries already existing). Please also try to avoid queries that write data. Stockmaniac-base has a VO/DAO object layer (FIXME: link) which provides a store() method for each object which should be used to write records.

Store custom queries in a separate library in your plugin directory. They should *not* be in the logic part of the code. Especially if the query is getting longer and/or has many variables.

The library should be called 'sql.inc.php' and loaded via require_once() on top of your plugin code.

Each query itself should be defined as a function which may or may not take additional parameters. The function name should start with 'sql_' followed by the nature of the query. That is either 'select', 'insert', or 'update'. At the end should be a hint on what the query is actually doing.

For example:

No matter if custom query or predefined query, the result can always be accessed through the sql_get* methods. There is:

• dbc::sql_get_result()
• dbc::sql_get_array_result()
• dbc::sql_get_indexedlist_result()
• dbc::sql_get_object_result()
• dbc::sql_get_objectarray_result()
• ... and maybe some more, take a look at the dbc class ...

Functions inside Plugins

Custom functions should be stored in a separate query library in your plugin directory. They should be kept as general as possible (as functions always should be).

The library should be called 'lib.inc.php' and loaded via require_once() on top of your plugin code.

A good example might be the UserManager: the objective of this plugin is to provide a interface to handle accounts. Of course it would be neat to have the ability to change passwords. And it would be even nicer if StockManiac would suggest random generated passwords. The logic of how to generate a random password is something that is important but not core logic of how to manage users. Thus it has been sourced out as create_password() and is stored in the lib.inc.php file.

Plugin Redirection Feature

This feature can be used in Editor plugins where the user submitted some data. For example Dividend payments or a new Order. StockManiac will automatically process the POST data and then call the redirect target. This is usually the corresponding Display. Thus, when data was submitted in the DividendEditor, the user will be redirected to the DividendDisplay. So he can see the changes immediatelly.

To use this in your plugin code simply set the redirect target plugin as a hidden input field. Like:

<input name="redirect" value="DividendDisplay" />

Sending Regular Messages to the User

Please use the stockmaniac::message() method for status messages you want the user to see. For example:

Note that all strings that will be shown to the user should be enclosed into a http://php.net/gettext() call (or _() for short). This is very essential for proper localization.

StockManiac will take care that everything is shown. You can call message() as often you like. All messages will be queued and printed the next time the page reloads.

Sending Error Messages to the User

If your code discovers a non-major error or you want to tell something important to the user please use the stockmaniac::error() method to do so. For example:

This will print your message to the user. In addition it tries to print all errors that may have been collected until that point. Finally it stops StockManiac right away.

Your code, as well as anything else, wont be able to continue any further. So please be carefull with this. It works like:

Plugin Specific Settings

Each plugin may define any number of settings. It should be done in the plugins 'init.php' file and look like:

Inside the plugin code settings can be accessed via the StockManiac object. Just like:

The get_setting methode will also accept the setting type ('plugin' or 'account') as second parameter. While 'plugin' is assumed as default type. As third parameter it accepts the name of a plugin. If the third paramenter is omitted, it will assume that $SM->EDITOR is the currently running plugin and use this name to locate the setting. For example:

To apply changes the user should always go through the 'Settings' plugin. As this is supposed to be the centralized place. Plugins itself should _not_ implement anything that allows a user to change preferences.

Account Specific Settings

Account Settings are defined in account_defaults.inc.php and are structured similar as plugin specific settings. As a consequence they are stored similar: in the stockmaniac::$_SETTINGS array. Just like the plugin specific ones.

account_defaults.inc.php defines reasonable defaults for all users. They can be changed by each user individual through the settings plugin.

StockManiac will load the defaults as well as changes (from database) during the login phase. The resulting array is then cached inside /each/ session.

Account settings may accessed via stockmaniac::get_setting() as well.

Data Types

Each settings record must be of a certain type. The type does actually regulate how the option field is interpreted by StockManiac. And how the Setting plugin will present it in the menu.

[string]

• regular text string

[bool]

• just true or false, where true=1 and false=0

[integer]

• unsigned integer

[select]

• a string value that is selected out the list specified in the 'values' element
• if a 'labels' element is present, its content will be show to the user for selection
• the 'val' element defines the default selection, it might be a string or a number (unsigned integer), whatever fits best
• the 'values' element defines all possibilities for 'val'
• the 'labels' element defines human readable labels for all 'values'
• labels can (and should) be localized
• note: the labels array length and order *must* match with the 'values' elements length and order!
• see the [range] setting example above

[list]

• a list of values seperated by '|'
• the user may edit each single value
• the user may add or remove values

Background of this Implementation

There is a array with all settings maintained inside the StockManiac object. This array is supposed to be private. While the get_setting* methods expose informations out of it. Usually for use inside a plugin. The array is structured like this:

The entire array is constructed in three steps during the plugin initialization phase:

• 1. gather the default values right from the init.php files
• 2. query the Setting table and apply all changes that the user or administrator has made to the defaults.
• 3. safe the final configuration in the current session so that steps one and two are not performed at each page reload

Please read schema.png (FIXME: link) for details on the Setting table design.

Design

Themes reside in themes/ directory

• one directory per theme
• themes are *pure* CSS, eventually some images, nothing else

separate CSS styles into multiple files

• one file for layout + one file for styling
• themes may use the general "master" layout styles *or* implement their own. This is the theme writers choice (it saves work to use the default layout).The themes class handles this and will fall back to the top-level layout.css file if the theme does not supply one
• finally it should be possible to re-design the *entire* application by writing a theme (css) file

utilize alternate style sheets

• CSS offers a "alternate style sheets" mechanism. StockManiac will automatically publish *all* themes that way. So users can select the look directly in the browser, whenever they feel like! ie. FireFox does this very well...
• all StockManiac does is setting the preferred (non-alternate) style, individually for each user, admins may set a system wide default in stockmaniac.inc.php

CSS Classes

All plugins are surrounded by div tags like

    <div id="plugin_<pluginname>"> ... </div>

More complex plugins may (in addition to id="plugin_<pluginname>") set a class to indicate which template is currently shown. The classname must be lowercase and be prefixed with 'template_'.

For instance the complete div tag surrounding the Asset plugin while the TimeTags template is in use would look like this:

    <div id="plugin_asset" class="template_timetags">
        ...
    </div>

Various items with "special" IDs exists as well, see the list below. Usually these are items of general nature which are only allowed to appear once per page... thus the ID. Items are prefixed with 'item_'. There is, for instance, one item on the main page called 'item_editor'. It is to group the editors head, the editor itself and the bar to select other editors since all three are logically related. The 'item_main' is quite similar.

Classes are used to identify "objects". For instance a news item, a document, a result table or a selection bar. Objects are usually constructions of several nested tags where only the surrounding tag is marked with the class name. Use descendend selectors to format the inside.

there are a number of "special" classes that could be set by display logic in template code (in fact its sort of a extension to CSS pseudo elements). For instance if the user clicked a item in a list then it may put in a marker that allows to highlight the item. These classes have verbal names such as "selected". Please do *not* assign generic styles to them.

all CLASSES and IDs must be LOWERCASE. (this is currently conflicting with plugin names but can be easily solved by applying the 'lower' filter in smarty)

Tags to be used in Templates

In general all HTML elements should be used the way they are supposed to be used. For example using <div> to emphasize a single word in a text *is* wrong. Inline elements such as <em>, <strong>, ... should be used for this purpose.

Try to avoid tags wherever possible. Less is better. CSS selectors are very powerful. Utilize them. Especially avoid nested <div> constructions. This *is* ugly. The goal is to have considerably less markup than content. StockManiac output should be seen as raw document text which can still be read by human beings.

Tables are *not* to be used for any formating or layout purposes. Do not put cellpadding, cellspacing, align, ... etc. attributes into tables. If using a table make sure that <caption>, <thead>, <tbody> and eventually <tfoot> are well defined.

Do not use tags and attributes deprecated in XHTML. Especially avoid all those formatting-only tags such as <font>, <b>, <i>, ... as all of this should be done in CSS only.

Do never put "style" attributes into any tag. Styles always go into CSS files. (There is actually one exception to this rule: the RiskPlugin uses style="..." to set the background color for the risk indicator. Thats because the actual color is chosen by ranking logic in the code. This should remain the one and only exception...)

Markup Structure in Templates

Markup code must be structured in a meaningful way in order to get the full power out of CSS selectors. Therefore stick to a few convetions:

• use <h1>, <h2>, ... tags for headings of any sort. For instance the plugin name on top or to indicate a section, etc...
• order all elements the way they are supposed to be read. Ie: CSS can select the first <p> that follows a <h1>. Thus the <p> should be related to the <h1>, not vice versa.

Special Class and ID Names

Special Class Names

• "selected": something the user explicitly selected by clicking it
• "profit": a number that is positive ("good") for the user, see "loss"
• "loss": a number that is negative ("bad") for the user. Note: any sort of cost such as charges are positive numbers in a mathematical sense, but still considered "bad" from a users perspective.
• "row1", "row2": those classes two are cycled through many listings so that row2 background differs from row1. (Isn't there a way to do this in CSS only?)
• "message", "notice", "error": these identify messages shown to the user. "message" is lowest importance, "error" is highest importance. All classes do/should not set any formating except for color and maybe font as its not sure where/when exactly those strings appear...
• "signal", "pivotalpoint", "uphit", "downhit": Very special, only used in QuoteRecords plugin to visually outline signals, pivotal points and up/down hits in the listing The correct classname for a record is figured out by the PHP code
• "emptyposition", "shortposition": those two are very special as they are currently bound to the Portfolio plugin only. I have no idea how else to visualize the position type. Thats too much logic for templates, but easy in the PHP code...
• "item_commentbox", "item_commentbox_content", "item_commentbox_marker": All together this makes opup like box with hidden content. When the mouse hovers over this box, the content will be shown. Use on any element you like. Important is that inside it is a

      <span class="item_commentbox_content">
  

  element holding the actual content. Then a

      <span class="item_commentbox_marker">
  

  element should be used somewhere close to indicate the presence of a hidden content box, for example:

      <a class="item_commentbox">
          <span class="item_commentbox_content">this is hidden content</span>
      </a>
      <span class="item_commentbox_marker"><!-- hover here --></span>
  

  where the <a> can be any other tag as well.

Special ID Names

• "item_menubar", "item_navigation", "item_main", "item_editor", "item_external": Group the plugins. Used for application wide "top level" layouting in layout.css No colors and stylings here please! It should be theme independent if possible.
• "item_login", "item_logout", "item_fatalerror": a page shown when StockManiac must stop because of some real weird error condition
• "item_pager": The pager used in various plugins. Its a class rather than a id since it may appear twice on the same page. For convennience the pager is a list and should always be created with stockmaniac::Pager()
• "item_infobox": A list containing end-user messages of varios importance it appears whenever there is someting the user should see/know
• "item_timetags": This ID marks a unified list containing time tag data. The list is special because widths are calculated in the plugin code and put in place by the template code. Each list element is supposed to match with those from the day listing.
• "item_inputwindow": Quick input forms that should be displayed on-top of everything else. Such "windows" are used whenever the user must create some new item. Therefore they should draw the users attention. This is best achieved by showing them in the middle of the screen with very distinctive colors such windows are for *input* only and should look alike throughout the entire application. id=item_inputwindow should be applied to 'fieldset' elements only
• "item_contentwindow": Similar to "item_inputwindow" except that content windows are used to show content. So they should never contain input forms. use them to show additional data that doesn't quite fit on the screen, the comments plugin is a good example. id=item_contentwindow should be applied to 'fieldset' elements only''
• "_debug_": Very special ID to format what the gui::debug() call prints. not necessary to define this in themes because all debug calls are removed before each release.

Design

The stockmaniac-base package provides two interfaces for unit testing: stockmaniac_webtestcase is to run web tests through the scriptable browser supplied by SimpleTest. stockmaniac_unittestcase is to run tests directly on classes or functions.

All else realted to tests resides in the gui/test/ directory.

To run the whole suite just access run_all.php through a browser (it can also run directly on the shell, have a look at the simpletest docs). To run partial tests check what other run_*.class.php files are available in test/.

The tests are run against a real database which is set up automatically by - right - a testcase. The file mysql-structure.sql will be used as testing schema. Connection details are specified in path.inc.class.php. (I am aware of SimpleTests "mock objects" but I also know that there have been many bugs because of discrepancies between code and schema and testdata vs. new code... etc... so I want to test the whole thing...).

Generally the unittesting in StockManiac is focused on testing end-user functionality with SimpleTest's WebTestCase and SimpleBrowser features (available via stockmaniac_webtestcase and stockmaniac_unittestcase). The reasoning behind this is simple: it will test all application layers, from the database via classes to display code to the template engine. That way we test large chucks of code at once with only a few simple tests (often as simple as sending a form and checking the resulting output).

Conventions

To keep all the testcases organized I made up a few conventions:

• all testcases reside in the test/ subdirectory
• testcase suite(s) should be called by a run_<desc>.class.php file (see below)

testcase files should be named like this:

run_<desc>.class.php

• run suite_* testsuite(s) that test <desc>
• accessable via browser
• cares about including the code (require_once(), __autoload(), etc...)
• should not contain any testcase code

suite_<desc>.class.php

a class subclassing and - perhaps - extending TestSuite
should call classtest_*, test_* and/or webtest_* testcases in its constructor

test_<desc>.class.php

• testcases to test *internal* behaviour of <desc> and/or specific tests for a known/reported bug
• should not contain any reporter code
• should extend from stockmaniac_unittestcase

classtest_<classname>.class.php

• testcases for class <classname>
• should not contain any reporter code
• should be self-sufficient (test only test <classname>, no other classes, no database stuff, etc...)
• should extend from stockmaniac_unittestcase

webtest_<pluginname>.class.php

• testcases for plugin <pluginname> that test the plugins user interface with SimpleTest's WebTestCase and/org SimpleBrowser features
• should not contain any reporter code
• should extend from stockmaniac_webtestcase
• a webtest_<pluginname>.class.php should exist for *each* plugin and should focus on testing the end-user functionality of the plugin, especially all input forms and the outcome after sending it (the past showed many bugs because of discrepancies in this area).

all other testcases (except webtest_<pluginname>.class.php files) should be written as needed, ie. when fixing a bug or refacturing something... etc...

as usual the first part of the filename (everything before the .class.php extension) must match with the class definition in the file. This is so the __autoload() mechanism will work.

Design

stockmaniacd is a multi-threaded daemon to handle Quote and News updates as well as math expression solving seemless in the background, without requiring user interaction and/or the gui part being present. All code is written in perl, using interpreter based iThreads.

The workflow inside the daemon is visualized in the following picture. Essentially it pushes workitem objects (FIXME: link) through its queues from thread to thread.

Each thread is implemented as its own class following this scheme:

Quote updates are handeled with finance-quote.sourceforge.net, News updates are done via XML::RSS and (later) XML::Atom. For math expression solving is Math::Expression::Evaluator being used.

Communication between GUI and Daemon is realized by utilizing the XML-RPC protocol as implemented in RPC::XML

in the future, stockmaniacd might do much more than just fetching data from the internet (ie email notifications on certian events, ... etc...)

Functional Specification

users should be able see the current update status

• when was a stock/news-feed updated last time
• which updates do not work (no quote found for symbol, ...)
• the vision is to use XML-RPC (not yet implemented)

users should be able to request a instant update

• perhaps for one or more stocks at once
• 'request' means the daemon should try to update as soon as possible, but should not interrupt in-progress ones
• we need flow control or priorities here
• the vision is to use XML-RPC (not yet implemented)

data retrieval in general

• must be able to handle huge quantities of 'work' if needed
• ie I have ~130 securities and ~20 news feeds in my production database right now, growing
• therefore data retrieval must be scalable in order to provide new data for each single stock or news feed 'in time'.
• this is achieved by using threads, where their number is subject to configuration
• data retrieval should have *no* impact on GUI performance, nor on the number of users in the system (ie we dont retrieve things per user - we only retrieve per stock or feed, no matter how many users trade/read a stock or feed)

quote updates

• consider business-hours ('when the market is open') to avoid producing tons of queries during the night...
• currently only on global time range can be specified
• in general, one per quote source would provide more flexibility
• provide features such as 'look up this symbol from this source, now'

NewsFeeds updates

• should be independend from quote updates
• respect TTLs as specified by each feed

configuration

• configuration file for administrative (thus more static) settings such as database login, thread allocations, ..., system security related things
• command line parameters for startup options (debug mode, verbosity, ...)
• runtime settings on per-user basis, stored in the database (ie 'send me e-mail notifys', ... , etc...)

communication with the web-application

• done using the XML-RPC protocoll
• needs some sort of authentication and/or access restriction as it will be vulnerable to DoS attacks, by nature.

status and logging facilities

• log general information in various levels
• provide features where users may request a current status (ie when a stock N was last fetched? was this successful? What was the error?), this is to be implemented via XML-RPC.
• for the log file default is to use syslog, however logging to a regular file should be possible (currently not implemented)

security

• >the daemon should use a restricted database login (just as fetch-objects.pl is doing right now)
>
• only a few tables must be read/written by the daemon

speed

• essential and resonable static information should be kept in memory
• ie Stock symbols, the time of each last fetched quote, Feed TTLs, Feed Item identifiers, ...
• ie avoid DB queries whenever possible

notifications (not yet implemented)

• since the daemon is fetching quotes, it knows first about any changes
• so it could notify the user whenever a buy/sell limit is hit or when there's news on a certain position, ... etc...
• not yet implemented

Implementation Details

Why perl iThreads?

• mainly because it exists quite a while now, is considered stable and is well documented
• moreover it's quite easy to use (ie no additional thread accounting code necessary), everything I need is just there.
• the only notable downside is that, by design, iThreads are not very fast and consumes quite a lot memory cause the perl interpreter gets copied for every thread
• therefore we run workers threads for as long as possible, maybe with some code to restart them once a day or so (configurable) - this will free up and re-allocate memory (hopefully)
• the number of workers is configurable, subsystems can be disabled entirely (that way one could run multiple daemons, each on a separate host, one handles news, the other one quotes, both access the same database - this is what I consider 'scalable')
• the daemon may print frequent status informations containing the queue sizes, threads, ... etc ..., this is useful to fine-tune the number of workers or to draw statistics
• see stockmaniacd-workflow.odg (FIXME: link)

Why RPC::XML

• well, its good and stable I guess. It seems to be the most activly developed XML-RPC implementation at present. And the most complete one too (I initially tried Frontier::Daemon - its not as feature rich).
• downside is that its not available in Fedora Core, so...
• use (or install from) the 'Dries RPM Repository' at http://dries.ulyssis.org/apt/packages/perl-RPC-XML/info.html
• search at http://rpm.pbone.net/
• install straight from CPAN (perl -e shell -MCPAN)

Data Transfer between Threads

• is handled with thread-safe Queues
• the current implementation is passing references to (shared) Workitem objects through Thread::Queue queues
• depending on the iThreads implementation this might or might not be effective. I am not sure. It, however, works with a standard perl distribution and reduces package dependencies to quite a minimum.
• a previous implementation (dropped in 0.10.0) used Thread::Queue::Any to pass data structures. I do not believe this was more efficient as Thread::Queue::Any is using Thread::Queue behind the curtain...
• see stockmaniacd-workflow.odg (FIXME: link)

Signaling to control Threads

• currently handled through the transfer queues by passing 'special' WORKITEM objects
• a 'special' workitem is actually a regular workitem object of the type _SIGNAL_, ie created like:

      use StockManiac::Workitem;
      my $signal = StockManiac::Workitem->new('_SIGNAL_', SIGTERM, ...);
  

  the workitem $signal is holding the systems SIGTERM in this case. It might be any integer number, so even custom signals are allowed (though, they should not conflict with standard posix signals). The threads must be aware and act on them.
• I am not sure whether this solution is good or bad. Mainly it is done that way because
  • it is natural in the sense that it does not require additional signaling infrastructure
  • I could not get Perls $thread->kill('SIG') mechanism to properly do what I want
  • I can not see a real disadvantage with that approach right now, so I just move along

Scheduler Design

• runs as thread
• on startup load everything 'non changing' into memory (ie lists of SIDs, Symbols and last available Quote, ...)
• all these informations are kept in StockManiac::Workitem objects, organized in a CATALOG (see Catalog Data Structure below)
• the scheduler figures *what* must be done and *when* by looking through the CATALOG
• the scheduler processes completed work returned by the workers and writes this back to the database (FIXME: this is something the Dispatcher could do)
• see stockmaniacd-workflow.odg (FIXME: link)

Dispatcher Design

• runs as thread
• receives workitems from all other threads and distributes them
• checks type of the work item and distribute it to the correct worker queue FUTURE: here is a good place to dock the Notification subsystem
• see doc/stockmaniacd-workflow.odg

Worker Design o each worker runs as thread - the number of worker threads of each kind is configurable - watch daemon status logs to fine-tune the numbers o receive workitem objects from Dispatcher - each workhorse class (Quote, News, ...) has a separate queue - workers block when the queue is empty o process workitem(s) - it is up to the workhorse to decide wheter to process *one* item at a time or to fetch 5, 10 (or more) items in a row and process them at once - the order in which workitems come in should not matter (!) o completed work is sent back to the dispatcher - through a queue - again: the order should not matter. o a worker must be stateless - we do *not* know if one worker will ever process the same workitem again, since the worker might be stopped or die or some other worker will receive the workitem... - therefore all information that is necessary for processing a workitem *must* be kept within the workitem (there are data storage methods provided by the StockManiac::Workitem class)

Listener Design o runs as thread o listen for XML-RPC requests from the GUI (or something else) o process the request, get answers by talking to all other parts of the daemon via the scheduler (StockManiac::Workitem objects as actually kind of a communication protocol too) o its a proper implementation which supports 'introspection' by default o a limitation is that the listener can process one request at a time. If two come in, one must wait. This appears to be no problem in real-life as answers are usually pretty quick.

Data Structures

There are two essential data structures used in stockmaniacd. One are WORKITEMs, the others are CATALOGs.

workitems

Workitems are small chunks of data (objects in fact). They (better: references to them) are passed through the queues from one thread to the other. The threads take them as input and do whatever they do with it's content, then hand them over to the next thread (queue)...

A single work item is a object created from StockManiac::Workitem. Like:

    $workitem = StockManiac::Workitem->new(
        dest          => 'quote',
        profession    => 'stock',
        task          => 'update',
        id            => 23,
        lastprocessed => 1166178567,     # unix timestamp
        frequency     => 36200           # seconds
        ...
    );

    $workitem->add_data(
        { 'somevar'=>0, 'someothervar'=>1, ... }
    );

    $workitem->add_raw_data(
        { 'rawvar'=>5, 'otherrawvar'=>10, ... }
    );

The first should *only* be used with data that has undergone sanitiy checks. So everything returned by get_data() can be considered 'save'. While the 'raw' functions are supposed to handle no sanitized data. Ie structures as received from the internet. Raw data should always be considered 'insecure'.

Further workers, scheduler, dispatcher, ... may do things with workitems:

    $workitem->is_due()               # is this item due for processing?
    $workitem->start_processing()     # this item is now 'in progress'
    $workitem->is_signal()            # is this a signal or regular work?
    $workitem->set_status(1);         # whatever we did succeeded
    $workitem->finished_processing()  # whatever we did is done now
    ...

catalog

Catalogs are rather big and currently kept in the Scheduler only (that might change in the not-too-far future). Anyhow a CATALOG is a collection of workitem objects. For instance the Scheduler is using a catalog data to figure what must be done next (ie which workitems are due for processing).

Technically speaking a catalog is a 'array of workitem objects'. Typically it looks like:

    @CATALOG_name = (
        { $WORKITEM_1 },
        { $WORKITEM_2 },
        ...
        { $WORKITEM_n },
        ...
    );

    @CATALOG_stocks