Conversions: keys and other types

Ok I'm going to post something here I wrote while working on converting Yapeal from old userID ApiKey system to new custom keys. I've only glanced at it over last couple weeks so it's a little outdated maybe on some points but should be interesting anyway and helpful I hope.

Hi all been doing some some thinking about all changes are going to be needed in
Yapeal because of the new custom keys that are coming to the Eve APIs. I also
decided to take a little time and do some brainstorming and see if there are any
other changes that could be made based on my mostly mental list of development
ideas and the list of suggestions I've receive from all of you on things you'd
like to see done with Yapeal as well.

Before getting into the ideas all this thinking has generated I'm going to do a
little review for us of how things work in Yapeal now so I can show how some of
the new ideas maybe possible now that weren't before and how some of the ideas
could greatly change how Yapeal works and it's scalability for large and small
projects while also giving it more versatility and maybe make it easier to use
in a wider range of applications.

Okay let's start with how Yapeal decided which APIs to get each time it runs. It
starts by making a list of 'sections' (account, char, corp, eve, map, server) by
looking for files in class/ with names starting with 'Section'. It then puts
them in a random order so any bugs in any one section that might crash Yapeal
can't end up totally blocking other sections. Using the list it check if the
section has been enabled at the Yapeal wide level by seeing if it's 'active'.
For each active section another list of 'allowed' APIs is made which is passed
to the class in class/Section*.php when it is called. The section class then
makes another 'available' list this time of the per API class files found in
class/api/ where the first part of their names match the current section. The
available list is then used to filter the allowed list to come up with a section
level active API list. At this point the different sections start splitting into
two different groups with account, char, and corp doing some extra stuff and
the eve, map, and server sections just check which APIs from the active list
have expired 'cachedUntil' times and Yapeal should now try to retrieve it from
the Eve API server. The account, char, and corp section also do this last
check but they also have a couple of more things they need to do first.

So the extra stuff that account, char, and corp sections need to do is make yet
another couple of lists of course. The first list is a list of the 'registered'
users (accounts), character, or corporations which are check to see if they are
'active' and then a new active 'owner' list for each user/char/corp is made.
This list is filtered by the section level active list to come up with the list
that is then checked to see if it has an expired 'cachedUntil' time like the
eve, map, and server sections did above.

After re-read the above I've realized I finally written some of the
non-techinical overview documentation that people have always been looking for
and asking where they can find it for a long time. The irony is I write it
without intending to just so I can propose making some changes that could end up
invaliding several parts of it depending on if people like some of the ideas
from below.

Now to understand parts of the ideas I have I will have to get a little more
into the technical side of how Yapeal implements some things in the account,
char, and corp sections but I will try to keep it as non-technical as I can so
the ideas come through hopefully without getting bogged down in all the details
that will be need to make them work.

First I'll go over here how the current API two key system works and how it can
cause many issues for CEOs and Directors in any corporations plus how the new
custom key system tries to improve on it. The current two keys can be thought of
as account level keys since each of them works for all of the characters and the
corporations those characters belong to without separating corporation access
from personal access. The first problem in the two key system that the new
custom key system tries to overcome is separating corporation level access from
personal access to the APIs for CEOs and directors. In the current system if one
of the directors or the CEO of a corporation gives someone their full API key
they are trusting that person with full access to all the APIs for the
corporation as well. They also have to trust people with access to their
personal data like assets etc so the other person could have access to the
corporation API data as well which made some of them unwilling to do so for some
applications that could have been very useful to the whole corp. Now they have
the option to just give someone access to their personal stuff like everyone
else has always had an option to do or just corp data without worrying about
what personal stuff they have also given away to someone. The custom key
system also lets you have more than two keys and you can have very fine grain
control over which APIs each key allows access to for both personal and
corporation keys. One of the things that was tried in the current two key system
to keep CEOs and directors from having to give away everything like above was to
let people with certain roles to also be given access to some of the corp APIs
but it was buggy so you couldn't use titles but always had to assign the roles
directly through the 'wonderful' in game interface that CCP provides. It wasn't
use much in most corporations because of that and other reasons but did allow
for some interesting Meta-gaming chances for people that choose to do so as part
of their game play on unsuspecting corps. The custom API system limits creating
corporation keys to only the CEO and directors which hopefully are a little more
security aware than say the people you give access to a POS so they can
refueling it or have pickup stuff from corp buy orders. So now that I've covered
the main differences between the old and new keys and the main improvements the
custom system gives us I'll next need to get into how Yapeal currently manages
keys and all the account, char, and corp APIs.

First we need to go into a little more detail about the 'registered' database
tables that are used to store and manage the keys plus the account (user), char,
and corp stuff so Yapeal knows how and which APIs to get. To make the following
a little easier to follow I will use the term 'owner' in place of 'user, char,
and/or corp' and probably also in a few places for 'account, char, and/or corp'.
As state above there are several 'registered' tables in the database and in fact
there is one for each type of 'owner'. In the 'user' table is stored the keys
and some info to let you determine which user you are talking about. The 'char'
table mostly has information about the characters but also a link to the
associated user as well. The 'corp' table has information about itself plus a
link to an associated character from 'char' table as well. The reasons they are
setup this way are that all 'owner' APIs require one of the API keys and the
'userID' (link) as parameters in the call to the Eve API server and the 'char'
and 'corp' ones also need a 'charID' (link) parameter. Some of the 'char' and
'corp' APIs also have other required and optional parameters but they aren't of
any concern to us for our needs and will be ignored. Since in the two key system
the keys are account level keys it makes the most sense to store them in the
'user' (account) table. By linking 'char' to 'user' and from 'corp' to 'char'
it's relatively easy to make sure the correct parameters get sent to the Eve API
server when new/updated data is requested. Now that the relationships between
the tables are understood which is the main reason used in determining the
current structure of all the owner tables we'll go over some of the
characteristics those relationships and structures force in how Yapeal works.

The first thing to notice is that both 'char' and 'corp' are dependent on 'user'
which is ok and done for good reasons but does put some limitations on the corp
stuff which isn't really absolutely required by the APIs. To highlight one of
the main problem let me use an example.

Lets say you have the information for the main character on someone's Eve
account in the 'user' and 'char' tables of Yapeal. Lets also have that character
be a director in a player corporation you have in Yapeal as well. So you have
their keys and user information in 'user' and one or more of the characters from
the account in 'char' included the director which links back to 'user' via
'userID'. You also have the corporation info in 'corp' which links to 'char' via
'charID'. Everything works great until one day the player decided he's tired of
doing all the extra work required of him as a director and is going to leave the
corporation to go do something else for a while and have some fun without all
the cares he has as a director. Now there is a problem since his info is needed
to get all the corp data in Yapeal from the Eve APIs. If there is another
director's or the CEO's info registered in Yapeal then it easy to fix by just
changing the link to the different charID and all is well but what if no one
else's info was in Yapeal? Still not hard to solve you just have to add one of
the remaining directors or the CEO to Yapeal and replace the charID info with
the new character's stuff. Does this sound like a mostly manual process to you?
It is by default in Yapeal. It could be automated at the application level but
as far as I know it hasn't been done fully automated because it requires lots of
calls to multiple classes inside Yapeal plus a fair amount of extra code in the
application to do. A system to update the corp info with a new existing
character that is started by an administrator as needed isn't to hard and I'm
sure has been done but automating it to detect these type of changes and doing
it on its own would be very very hard.

Another example would be where someone decides to sell one of their characters
and both the user and corporation info has to be changed. CCP makes this example
a little easier because the characters can't have any roles in the corporation
at the time of transfer I understand but it'll still cause some problems at the
character and user level. Now the changes to Yapeal as I foresee them wouldn't
include a way to directly do any of this either but it should require a lot less
code and should change automating it from a very very hard problem to just a
very hard problem which might be enough for someone to try to do it at least.

Now that we have at least a little better understanding of how the registered
tables work the question becomes how could they be changed to better take
advantage of the new custom keys and all the other changes CCP has made to the
APIs so they work with the custom keys. One thing that has change is the keys
aren't 'owned' by an account any more but by characters. You can still make keys
that work for all the characters on an account but the relationship between the
account (userID) and a key is much less important now and largely gone for all
but the 'all' keys. Yapeal doesn't really care about the 'owner' on a key it's
only really interested in which APIs any one key gives it access to. The only
reason it have to know anything beyond what APIs a key gives access to is so it
can make sure to include the proper 'ownerID' when storing the retrieved data
into the database. The only other place it really needs to associate something
to the key is in the case of 'all' keys because the API server needs to know
which character we are using it for on the 'char' APIs. I was one of several
developers asking CCP to keep the idea of 'all' keys alive because I thought it
would make things easier but now after working with the changed APIs I wonder if
we really need them. Okay since keys can be associated with 1-3 characters or a
corporation/character combination and the unlimited number of keys per available
having the keys stored in the 'user' table doesn't work anymore and moving them
to 'char' does work either. The keys also now have the associate vCode which is
per key. The best thing to do is have a new 'registered' table that stores the
keys and their vCode together. There also is a new API to find out additional
information about the keys account/APIKeyInfo which lets us know things like
which type of key it is 'Character'|'Corporation' and you can figure out
indirectly from it if the key is an 'all' key as well but currently not get a
full list of all the characters associated with them.

Programmers wanted

Just thought I'd let the couple people that actual look at this blog know I am looking for programmers to help with Yapeal's development. If you're interested contact me via owner info on project.

Yapeal, memory, and XML part 3

Hi all sorry it took so long before getting to this post but while writing the last one I got some ideas and decided to try them out in Yapeal.

Ok Let's do a summary from the other posts here. First there's several ways to work with XML in PHP. Some work with the whole document at a time like DOM, SimpleXML, and most versions of XSL. Others work with small pieces like SAX and XMLReader.

Next we know some APIs like Account Balance are small, some are large but limited like Wallet Journal, and others can become very large and aren't limited like Asset List. We also know that as Yapeal now stands it's doing a lot of converting and copying and duplicating of the XML which multiples the memory used. When Yapeal was first started it mostly did the small, or large limited APIs so memory never become much of a problem and SimpleXML made it just that, simple. Since then Yapeal has gone through a lot of changes but largely the logic used to process the XML hasn't changed even after adding some of the largest unlimited APIs. PHP has also gone through some changes in PHP 5.3 and it's now showing the memory use for some extensions that was hidden in prior versions. Putting all those things together with my ability to only test it with a small number of accounts at a time (< 5 and often only a couple) it's not to surprising that when we did some testing with a few hundred accounts there were a few issues. The biggest one was it used around 128MB of memory and there were few other issues with the ordering of the APIs  which let earlier APIs keep later ones from having a chance to get their data. I'm not going to get into the ordering problems here but I do plan on covering that in a future blog.

One of the great things with SimpleXML as well as DOM and XSL is you have XPath to help you cut out small part of well designed XML like a skilled surgeon with a scalpel but because of some poor design in many of the APIs IMHO trying to use it is more like trying to hack off of a piece of meat from a charging wild animal with a dull broad sword without get trampled.  Often times in Yapeal trying to use it ends up doing little more than cutting off the head and maybe getting it cut up into quarters that still have the skin on them. As I said XPath can let you get small parts but it actually does this in SimpleXML and DOM by making a copy or at least you end up having to do so yourself to use the result which when added to the XML design on the larger APIs ends up being a problem. Now just to make it clear I really like XPath, SimpleXML and most of the larger APIs actually don't have many of the bad design issues as some of the other APIs but it's very hard to work with them without using a lot of memory.

So give the above the question becomes would one of the other extensions work better in Yapeal? The DOM would have the same problems as now but be harder to work with. XSL if it used a SAX type backend would work but the one in PHP isn't so it's not going to be helpful either. SAX could improve the memory issues but it very hard to use and I believe it would make Yapeal un-maintainable and I don't see converting over to it. So that leaves just two extensions and only one of them is made for reading XML and not making it.

Let's look at XMLReader some more. It uses much less memory than SimpleXML, DOM, or XSL because it only deals with the XML in small pieces and it's easier to use than SAX. So far it sound like it could be a better fit for Yapeal but "the true is in the code" so to say. Just to keep this blog from getting any longer than needed I'll say after working with XMLReader for almost two weeks now that it does seem to be a better fit for Yapeal. So far in trying it with some other changes I've made I've seen memory use drop to less than half what it was before.

Now the only other thing that may be a problem but shouldn't be is do most hosting sites include XMLReader? They should because it's a standard included and enabled extension for PHP since version 5.1.0 but as I've found out after releasing Yapeal some hosting sites don't even have SPL (Standard PHP Library) available. So I'd like to hear from some people to get some idea how common it is or what they needed to do to get their host to make it available. For anyone that want to check for it try this:
php -r 'if (extension_loaded("xmlreader")) {print "YES!" . PHP_EOL;};'

Other than needing a different extension this change shouldn't be visible outside of Yapeal itself in any way but some of the other changes to work around the issues with the order in which Yapeal does the APIs may be more visible as they may require so database changes in the util* tables that may have to be done manually. If it does I'll write the instructions up with the SQL and post them.

That's it for now see you down the blog.

Yapeal, memory, and XML part 2

In the last blog post I covered some background on extensions available in PHP to work with XML. In this one I'll be look at the Eve APIs and how much memory they use in a very general way. I'll also cover how they effect Yapeal memory usage as a result. Let's get to it.


Most of the Eve APIs return a well defined amount of data like your current account balance or a list of things like the implants in your current clone etc. In the case of things like implants or skills the list can vary in length from 0 to some max number (10 for implants and currently 390 for skills) but there is a limit that can be figured out and planned for. These limits are generally small and take up little in the way of memory no matter which of the XML extensions is used. Next you have APIs like Wallet Journals and Transactions which can be consider large but still limited since they return at most 1000 entries at a time. Finally there's stuff like Asset List, Standings, and most of the corporation ones that even if they aren't truly unlimited their limits are much higher and for our proposes it will be assumed they can become very large. As an example recently while working with an application developer trying out Yapeal who had a list of a few hundred API keys we saw memory use climb to 128MB or more. That doesn't sound like much but when added to the memory needed by MySQL and web server etc on a shared host it does become a problem and caused Yapeal to run very slowly. Given that CCP is unlikely to change the APIs just for Yapeal it needs to be changed so it uses less memory and hopefully lessen the effects cause by a large amount of API data.

Now let's take a quick look at how Yapeal works with the API data. First it looks for a cached copy either in files on the harddrive or in DB cache table if they don't exist or it's past the cachedUntil date found in the copy it has it decided to try getting a fresh copy uses cURL from the API server or a configured proxy. The XML is first returned as a string then converted to a simpleXML object. With simple rowset type APIs the data is convert first to an associative array which is then used to built a SQL 'upsert' (combined insert on duplicate update) query. For more complex APIs the XML is converted into multiple arrays and upserted to one or more tables. You notice that there are times when there are up to 3 copies of the same data existing side by side. To make it worse there are actually times when there are at least other partial additional copy cause by limitations in the ability to separate out only the part needed while building the arrays in complex APIs. From the above it is clear that if any of the large or very large APIs are complex the memory requirement can grow very fast and it actually multiples inside Yapeal in most cases.

Ok now that we understand in general the types of APIs, their memory needs, and some of the bad effects they can have in Yapeal we have a good basis on which to start look at which XML extensions from part 1 would be best match to the need of Yapeal. Since doing so will take a while (read several more paragraphs) I'll leave that to the third and probably last part in this series.

Yapeal, memory, and XML part 1

Ok as everyone that been using Yapeal for a while knows it has become a bit of a memory hog of late. Partly this has been caused by adding some APIs that can be very large but some of it has been because of an early design decision to use simpleXML. Don't get me wrong I love simpleXML and I might not have even start on the project that grow into Yapeal without it but there's some things about it that are now cause issues. Let me take a little deter into the options available to work with XML in PHP in this post so everyone has some background.

• DOM - Been around for a long time and was design by the same people that did standard for XML, HTML, etc. It's based on seeing the XML document as a tree of elements with attributes and is considered OOP but not being made for PHP originally it object model is somewhat different. It is very powerful and there are many things it can do that none of the other options have but it's also hard to use and the tree it builds has to all fit in memory which start to become a problem with large documents. You can find DOM libraries for most programming languages on most OS platforms which makes it something everyone should learn and understand and know how to use if you are going to work with XML a lot.
• simpleXML - It was design as an easier to understand and use option to using DOM in PHP. It is an OOP design like the DOM but a better fit with the object model use in PHP since it was developed for it. It originally only allowed the document to be read but writing has since been added and you can use it to make new documents as well. You can do almost everything with it that you can with DOM but for creating complex documents etc it's usually better to use DOM instead. You can find implementation of it in other languages now but it's mostly use in PHP. It uses an in-memory tree like DOM does because they share a common low level library. Because of this passing a document between them is easy and is sometimes used to help fill hole where simpleXML can't do stuff that DOM can or where it's easier in DOM or the other way around.
• SAX - It's been around longer then the DOM has and instead of viewing XML as a document and making it into a tree it views it as a bunch of pieces or a stream of them and fires 'events' that have to be handled for each of them. It is not OOP but has only a function based interface to the XML. With SAX you can only go forward through the XML and the programmer have to keep track of where they are in the document and making sense out of all the 'events' is totally up to them. Most programmer find it very difficult to work with and once DOM became available most of them switched because of this. One big advantage to SAX is that it uses very little memory normally making it useful where either the target platform has very little or the document is very large and can't fit in memory. SAX can't be used to make XML documents only read them. It is usually available as a library for all platforms and programming languages.
• XSL - Used to transform XML documents into either another type of XML document or some other type of text output. The stylesheet used to do the transform is itself a XML document. XSL is great for converting XML to HTML or a CSV file but finds little application outside of that. It's based on DOM as well with many of the same issues with memory though there have been implementations based on SAX that allow for documents larger than memory.
• XMLReader - How I've come to think of it is like simpleXML for SAX. Instead of having to react to a bunch of 'events' with no context you iterate through a sequence of nodes (objects). Like SAX it have a small memory footprint since it only view the document in small pieces but because they are objects they help the programmer by provided more information than just something happened like in SAX. You can check if the node is empty or contents  something else, if it has attributes and their values, etc. All of these thing and others make using it and keeping track on where you are in the document as a programmer much easier.

• XMLWriter - It is a hybrid extension that let's you use either function or OOP access to creating XML documents. It's a compliment to XMLReader and fits some where between simpleXML and the DOM in complexity.

Ok that the end of this post in the next one I'll cover the different kinds of Eve APIs and how they differ mostly in the amount of memory each uses.

Experimenting with Yapeal blog

Ok I've decide to try a blog instead of just using the eve Yapeal thread and wiki for stuff. I've never done one before or even really followed anyone else's so no telling how much or what will end up here or if I'll continue to use it but had some ideas that I thought might be better serviced using a blog vs something else.

One thing I was thinking of use it for is a place to post some of my design ideas and about any changes I'm thinking about make to Yapeal. By using a blog maybe people will be more willing to add feedback in the comments.

I might also use it to post a few tutorials about Yapeal or programming in general.

Anyway that's some of my ideas and I'm very open to any other ideas people might have about what they would like to see here.