One giant commit: - Divided up servers and listeners per protocol. - Servers and Listeners no longer derive from Santiago. - Added protocol-based servers and listeners.

cfg.sample.py

@@ -8,6 +8,8 @@ user_db = os.path.join(data_dir, "users")
 status_log_file = os.path.join(data_dir, "status.log")
 access_log_file =  os.path.join(data_dir, "access.log")
 users_dir = os.path.join(data_dir, "users")
+santiago = os.path.join(data_dir, "santiago.sqlite3")
+
 
 product_name = "Plinth"
 box_name = "Freedom Plug"

ugly_hacks/santiago/README.rst

@@ -47,6 +47,10 @@ for the service.
 In a nutshell, that's the important part.  There are additional details to
 manage, but they're implied by and built on the system above.
 
+Each Santiago process is responsible for managing a single key and set of
+friendships, so multiple Santiagi per box (each with a different purpose or
+social circle) is completely possible and intended.
+
 Our Cheats
 ----------
 
@@ -98,6 +102,8 @@ Each node contains two dictionaries/hash-tables listing (1) what they serve and
 who they serve it to, and (2) what services they use, who from, and where that
 service is located.
 
+These are stored in the "Santiago" database, as three individual tables.
+
 What I Host and Serve
 ~~~~~~~~~~~~~~~~~~~~~
 
@@ -108,9 +114,9 @@ These data are stored as pair of dictionaries:
 - The GPG ID to Service dictionary.  This lists which service each user is
   authorized for::
 
-      0x0928: [ "proxy": "proxy", "wiki": "wiki",
+      0x0928: { "proxy": "proxy", "wiki": "wiki",
-                "drinking buddy": "drinking buddy" ]
+                "drinking buddy": "drinking buddy" }
-      0x7747: [ "wiki": "wiki", "proxy": "restricted_proxy" ]
+      0x7747: { "wiki": "wiki", "proxy": "restricted_proxy" }
 
 - The Service to Location dictionary.  This lists the locations each service
   runs on::
@@ -249,6 +255,25 @@ Note that:
 - Irrelevant signatures (intermediate links in the WOT) are stripped, hiding the
   WOT's structure from friends.
 
+Anachronisms
+~~~~~~~~~~~~
+
+It's odd because this has a potential for a number of irrelevant communications.
+
+It's possible for A to send multiple requests to B and for B to receive multiple
+requests before A acknowledges responses.
+
+A -> B, A -> B, B -> A, A -> B.
+
+Code Structure
+--------------
+
+Yeah, I really need to hammer this part out.  Stupid MVC model.
+
+So, listeners receive responses and pass them up to the controller that queues
+it for the responder.  Lots of listeners, a single responder.  Listeners have a
+single method, while responders have multiple (per type of response?).
+
 Unit Tests
 ==========
 
@@ -383,6 +408,52 @@ Functional Questions
     messages should be queued at a file-level so that each process who needs
     access can have it.
 
+:Santiago Updates: Updates are tricky things.  They're when we're most
+    vulnerable.  The question becomes: since both boxes need to know where they
+    are to communicate successfully, but at least one box may have changed its
+    location (even its Santiago), how do we handle those updates, while reducing
+    the vulnerability as much as possible?  Let's assume that A (the requester)
+    changes its locations frequently, while B (the server) does not.  A requests
+    a service from B and B then needs to reply.  How does B know where to reply?
+    It has a few old Santiago ports left over in the database.  A might also
+    have sent Santiago updates with the request message.  How does B handle
+    those updates?
+
+    Does B queue those Santiagi last in the update queue, are they checked
+    first, or is appending Santiagi not allowed?  Each creates a different
+    vulnerability.
+
+    If A's key is compromised, but his box is not, then the request is fake and
+    so are the new Santiagi.  The old ones are still valid.
+
+    If A's box is compromised, then his key is probably compromised too, and all
+    existing Santiagi are compromised.  This could be A trying to transition to
+    a new box without changing keys, though, so the new Santiagi are valid.
+
+    If A NAKs B's update message when A didn't ask for it, causing B to consider
+    that request from A (and the related Santiago) compromised, then that too
+    could be used by adversaries with a compromised key to deny A service.
+
+    What a bloody circle.  Both options are bad, but some worse than others?
+
+    Well, if we prevent Santiagi updates in messages altogether, B might never
+    find A again, if A moved.  So that sucks.  But, that's also overloading
+    messages and implicitly allowing push-updating.  If we allow pull-updating
+    only, then both boxes need to be accessible to one another at all times.
+    More secure, but a *lot* less useful.
+
+    Is it meaningful to consider some forms of signed communication more
+    vulnerable than others, or are we just saying that if the communication is
+    successfully signed, then it must be valid, damn the consequences?  I think
+    so, actually.  Otherwise, we start jumping at shadows.  There's no way to
+    know whether a key's been compromised until the revocation certificate is
+    deployed, and I can't verify anybody else's security measures.  Perhaps your
+    definition of security is "this key I share between me, my wife, our three
+    kids, and the dog's neighbor."  If I happen to trust the dog's neighbor
+    (but, oddly, not the dog itself), then I might trust the key.  If I don't
+    trust the second of three kids, then why am I trusting the key?  Trust is an
+    annoyingly deep subject, and one of the few good uses of the word "faith."
+
 References
 ==========
 

ugly_hacks/santiago/TODO

@@ -12,6 +12,8 @@
 - Spec out the system through test harnesses.  If the tests can run
    the system, it's complete.
 
+- Differentiate backends and the independent system.
+
 * Message Queuing
 
 - Process a maximum of X MB over Y requests per unit time Z per
@@ -22,3 +24,16 @@
 * Process Separation
 
 - Listeners aren't senders aren't controllers.
+
+* Actual PGP sig verification.
+
+* Send replies to the recipient's Santiago.
+
+- Queue the Santiagi and wait for a reply.  If we time out before
+   receiving a reply, go on to the next Santiago.
+
+* Move Santiago data store to James's databases.
+
+* Implement real Santiago request/reply/response.
+
+- Need to move to James's database first.

ugly_hacks/santiago/fbx/known_services.py

@@ -1 +0,0 @@
-{ "plinth": { "james": [ "http://192.168.0.12:8080" ], }, }

ugly_hacks/santiago/fbx/serving_to.py

@@ -1,2 +0,0 @@
-{ "james": { "wiki": "wiki" },
-  "ian": { "web": "plinth" }, }

ugly_hacks/santiago/fbx/serving_what.py

@@ -1,2 +0,0 @@
-{ "plinth": "http://192.168.0.13:8080",
-  "wiki": "http://192.168.0.13/wiki", }

ugly_hacks/santiago/fbx/settings.py

@@ -1,5 +0,0 @@
-{ "me": "nick",
-  "socket_port": 8080,
-  "max_hops": 3,
-  "proxy_list": ("tor")
-  }

ugly_hacks/santiago/protocols/http.py

@@ -0,0 +1,87 @@
+import cherrypy
+import santiago
+from simplejson import JSONEncoder
+
+encoder = JSONEncoder()
+
+
+# dirty hacks for Debian Squeeze (stable)
+# =======================================
+
+def fix_old_cherrypy():
+    """Make Squeeze's CherryPy forward-compatible."""
+
+    for y in range(0,3):
+        for x in range(0, 7):
+            print "WARNING",
+        print ""
+
+    print("You're using an old CherryPy version!  We're faking it!")
+    print("Expect the unexpected!  Raar!")
+
+    def jsonify_tool_callback(*args, **kwargs):
+        response = cherrypy.response
+        response.headers['Content-Type'] = 'application/json'
+        response.body = encoder.iterencode(response.body)
+
+    cherrypy.tools.json_out = cherrypy.Tool('before_finalize', jsonify_tool_callback, priority=30)
+
+if cherrypy.__version__ < "3.2":
+    fix_old_cherrypy()
+
+
+# actual HTTP Santiago classes.
+# =============================
+
+class SantiagoHttpListener(santiago.SantiagoListener):
+    """Listens for connections on the HTTP protocol."""
+
+    DEFAULT_RESPONSE = """\
+<html><head><title>Use it right.</title></head><body>
+
+    <p>Nice try, now try again with a request like:</p>
+    <p>http://localhost:8080/santiago/(requester)/(server)/(service)</p>
+
+    <dl>
+        <dt>requster</dt><dd>james, ian</dd>
+        <dt>server</dt><dd>nick</dd>
+        <dt>service</dt><dd>wiki, web</dd>
+    </dl>
+
+    <p>This'll get you good results:</p>
+    <code><a href="http://localhost:8080/santiago/james/nick/wiki">
+        http://localhost:8080/santiago/james/nick/wiki</a></code>
+
+    <p>See the <code>serving_to</code>, <code>serving_what</code>, and
+    <code>me</code> variables.</p>
+
+</body></html>"""
+
+    def __init__(self, instance, port=8080):
+        super(SantiagoHttpListener, self).__init__(instance)
+        self.socket_port = port
+
+    @cherrypy.expose
+    @cherrypy.tools.json_out()
+    def serve(self, key=None, service=None, server=None, hops=3, santiagi=None):
+        """Handles an incoming request."""
+
+        return super(SantiagoHttpListener, self).serve(
+            key, service, server, hops, santiagi)
+
+    @cherrypy.expose
+    def index(self):
+        """Do nothing, unless we're debugging."""
+
+        if santiago.DEBUG:
+            return self.DEFAULT_RESPONSE
+
+
+class SantiagoHttpSender(santiago.SantiagoSender):
+    """Responsible for answering HTTP requests."""
+
+    @cherrypy.tools.json_out()
+    def proxy(self, key, service, server, hops=3):
+        """Forwards on a request to another Santiago."""
+
+        return super(SantiagoHttpSender, self).proxy(key, service, server, hops)

ugly_hacks/santiago/santiago.py

@@ -20,178 +20,149 @@ or later.  A copy of GPLv3 is available [from the Free Software Foundation]
 
 """
 
-# If you're crazy like me, you can turn this script into its own
+# debug hacks
-# documentation by running::
+# ===========
-#
-#     python pylit.py -c backup - | rst2html > backup.html
-#
-# You'll need PyLit_ and ReStructuredText_ for this to work correctly.
-#
-# .. _pylit: http://pylit.berlios.de/
-# .. _restructuredtext: http://docutils.sourceforge.net/rst.html
-#
-# .. contents::
 
-import cherrypy
+DEBUG = 1
-import os
-from simplejson import JSONEncoder
-
-
-DEBUG = 0
-encoder = JSONEncoder()
-
-# random setup tools
-# ==================
 
 if DEBUG:
-    for x in range(0, 3):
+    """A few hacks to make testing easier."""
-        for y in range(0, 7):
+
+    def cfg_hack():
+        import sys
+        sys.path.append("../../")
+        import cfg
+
+    def ohnoes():
+        for y in range(0, 3):
+            for x in range(0, 7):
                 print "WARNING",
             print ""
         print "You're in DEBUG MODE!  You are surprisingly vulnerable!  Raar!"
 
-def fix_old_cherrypy():
+    ohnoes()
-    """Make Lenny's CherryPy forward-compatible."""
+    cfg_hack()
-
-    for x in range(0,3):
-        for y in range(0, 7):
-            print "WARNING",
-        print ""
-
-    print("You're using an old CherryPy version!  We're faking it!")
-    print("Expect the unexpected!  Raar!")
-
-    def jsonify_tool_callback(*args, **kwargs):
-        response = cherrypy.response
-        response.headers['Content-Type'] = 'application/json'
-        response.body = encoder.iterencode(response.body)
-
-    cherrypy.tools.json_out = cherrypy.Tool('before_finalize', jsonify_tool_callback, priority=30)
-
-if cherrypy.__version__ < "3.2":
-    fix_old_cherrypy()
 
 
-# actual Santiago
+# normal imports
-# ===============
+# ==============
+
+from collections import defaultdict as DefaultDict
+import util
+
 
 class Santiago(object):
     """Santiago's base class, containing listener and sender defaults."""
 
-    DEFAULT_RESPONSE = """\
-<html><head><title>Use it right.</title></head><body>
-
-    <p>Nice try, now try again with a request like:</p>
-    <p>http://localhost:8080/santiago/(gpgId)/(service)/(server)</p>
-
-    <dl>
-        <dt>gpgId</dt><dd>james, ian</dd>
-        <dt>service</dt><dd>wiki, web</dd>
-        <dt>server</dt><dd>nick</dd>
-    </dl>
-
-    <p>This'll get you good results:</p>
-    <code><a href="http://localhost:8080/santiago/james/wiki/nick">
-        http://localhost:8080/santiago/james/wiki/nick</a></code>
-
-    <p>See the <code>serving_to</code>, <code>serving_what</code>, and
-    <code>me</code> variables.</p>
-
-</body></html>"""
-
     def __init__(self, instance):
-        self.load_instance(instance)
+        """Initializes the Santiago service.
 
-        # TODO Does the listener need to know what services others are running?
+        instance is the PGP key this Santiago service is responsible for.
-        # TODO Does the sender need to know what services I'm running?
+
-        self.load_serving_to()
+        Each service contains one or more senders and listeners, primarily
-        self.load_serving_what()
+        divided by protocol, all pulling from and adding to the same pool of
-        self.load_known_services()
+        services.
+
+        Each Santiago keeps track of the services it hosts, and other servers'
+        Santiago services.  A Santiago has no idea of and is not responsible for
+        anybody else's services.
+        
+        """
+        self.instance = instance
+        self.hosting = self.load_dict("hosting")
+        self.keys = self.load_dict("keys")
+        self.servers = self.load_dict("servers")
+
+        self.listeners = list()
+        self.senders = list()
+
+        # load settings by name
+        settings = self.load_dict("settings")
+        for key in ("socket_port", "max_hops", "proxy_list"):
+            setattr(self, key, settings[key] if key in settings else None)
+
+    def load_dict(self, name):
+        """Loads a dictionary from file."""
+
+        # FIXME: figure out the threading issue.
+        #return util.filedict_con("%s_%s " % (cfg.santiago, self.instance), name)
+        return {
+            "hosting": DefaultDict(list),
+            "keys": DefaultDict(list),
+            "servers": DefaultDict(lambda: DefaultDict(list)),
+            "settings": DefaultDict(None)
+            }[name]
 
     def am_i(self, server):
-        return str(self.me) == str(server)
+        """Hello?  Is it me you're looking for?"""
         
-    def load_instance(self, instance):
+        return self.instance == server
-        """Load instance settings from a file.
 
-        A terrible, unforgivable hack.  But it's a pretty effective prototype,
+    # Server-related tags
-        allowing us to add and remove attributes really easily.
+    # -------------------
+
+    def provide_at_location(self, service, location):
+        """Serve service at location.
+
+        post::
+
+            location in cfg.santiago.hosting[service]
 
         """
-        settings = run_file("%s%ssettings.py" % (instance, os.sep))
+        self.hosting[service].append(location)
-        for key, value in settings.iteritems():
-            setattr(self, key, value)
 
-        self.instance = instance
+    def provide_for_key(self, service, key):
+        """Serve service for user.
 
-    # FIXME I need to handle instance vs controller correctly.  this is the
+        post::
-    # wrong place.
-    #
-    # I'm putting instance data into the controller, which is nutters.  Too
-    # tired to fix tonight, though.
-    #
-    # These data should probably be loaded in the server and listener,
-    # respectively, but I don't know whether one needs to know about the other's
-    # services.
 
-    def load_serving_to(self):
+            service in cfg.santiago.keys[key]
-        """Who can see which of my services?"""
-
-        self.serving_to = run_file("%s%sserving_to.py" % (self.instance,
-                                                          os.sep))
-    def load_serving_what(self):
-        """What location does that service translate to?"""
-
-        self.serving_what = run_file("%s%sserving_what.py" % (self.instance,
-                                                              os.sep))
-    def load_known_services(self):
-        """What services do I know of?"""
-
-        self.known_services = run_file("%s%sknown_services.py" % (self.instance,
-                                                                  os.sep))
-    @cherrypy.expose
-    def index(self):
-        """Do nothing, unless we're debugging."""
-
-        if DEBUG:
-            return DEFAULT_RESPONSE
-
-
-class SantiagoListener(Santiago):
-    """Listens for requests on the santiago port."""
-
-    def __init__(self, instance, port=8080):
-        super(SantiagoListener, self).__init__(instance=instance)
-
-        self.socket_port = port
-
-    @cherrypy.expose
-    @cherrypy.tools.json_out()
-    def santiago(self, from_id=None, to_id=None, service=None, hops=0): #, new_santiago_id=""):
-        """Handles an incoming request.
-
-        FIXME: handle new Santiago ID list.
 
         """
-        message = { "requester": from_id,
+        self.keys[key].append(service)
-                    "server": to_id,
-                    "service": service,
-                    "hops": hops, }
-                    #"santiago": new_santiago_id }
 
-        # FIXME: this is being dumb and not working how I expect it.  later.
+    # client-related tags
-        # if not self.i_am(message["server"]):
+    # -------------------
-        #     return self.proxy_santiago_request(message)
 
-        try:
+    def learn_service(self, service, key, locations):
-            return self.serving_what[
+        """Learn a service to use, as a client.
-                self.serving_to[message["requester"]][message["service"]]]
-        except KeyError:
-            # TODO: handle responses.  should a fail just timeout?
-            return None
 
-    @cherrypy.tools.json_out()
+        post::
-    def proxy_santiago_request(self, message, hops=3):
+
+            forall(locations, lambda x: x in cfg.santiago.servers[service][key])
+
+        """
+        self.servers[service][key] += locations
+
+    def consume_service(self, service, key):
+        return self.servers[service][key]
+
+    def add_listener(self, listener):
+        """Registers a protocol-specific listener."""
+        
+        self.listeners.append(listener)
+
+    def add_sender(self, sender):
+        """Registers a sender."""
+
+        self.senders.append(sender)
+
+    # processing related tags
+    # -----------------------
+        
+    def serve(self, key, service, server, hops, santiagi):
+        """Provide a requested service to a client."""
+
+        if santiagi is not None:
+            self.learn_service("santiago", key, santiagi)
+
+        if not self.am_i(server):
+            return self.proxy(key, service, server, hops=hops)
+
+        if service in self.keys[key]:
+            return self.hosting[service]
+
+    def proxy(self, key, service, server, hops=3):
         """Passes a Santiago request off to another known host.
 
         We're trying to search the friend list for the target server.
@@ -203,20 +174,27 @@ class SantiagoListener(Santiago):
         if (hops < 1):
             return
 
-        # this counts as a hop.
         hops -= 1
 
-        # TODO pull this off, another day.
+        # TODO pick the senders more intelligently.
-        return str(message)
+        return self.senders[0].proxy(key, service, server, hops)
 
 
-class SantiagoSender(Santiago):
+class SantiagoListener(object):
-    """Sendss the Santiago request to a Santiago service."""
+    """Listens for requests on the santiago port."""
 
-    def __init__(self, instance, proxy):
+    def __init__(self, santiago):
-        super(SantiagoSender, self).__init__(instance=instance)
+        self.santiago = santiago
 
-        self.proxy = proxy if proxy in self.proxy_list else None
+    def serve(self, key, service, server, hops, santiagi):
+        return self.santiago.serve(key, service, server, hops, santiagi)
+
+
+class SantiagoSender(object):
+    """Sends the Santiago request to a Santiago service."""
+
+    def __init__(self, santiago):
+        self.santiago = santiago
 
     def request(self, destination, resource):
         """Sends a request for a resource to a known Santiago.
@@ -231,7 +209,8 @@ class SantiagoSender(Santiago):
         - Other Santiago listeners.
         - An action.
 
-        post:
+        post::
+
             not (__return__["destination"] is None)
             not (__return__["service"] is None)
             # TODO my request is signed with my GPG key, recipient encrypted.
@@ -239,24 +218,64 @@ class SantiagoSender(Santiago):
         """
         pass # TODO: do.
 
+    def nak(self):
+        """Denies a requested resource to a Santiago.
 
-# utility functions
+        No reason is given.  All the recipient knows is that the host did not
-# =================
+        have that resource for that client.
-
-def run_file(filename):
-    """Returns the result of executing the Python file.  Terrible idea.  Effective
-    hack.
-
-    TODO: Replace this with James's database stuff!
-
-    If you try to use this in the wild, I will hunt you down and cut you.
 
         """
-    with open(filename) as in_file:
+        pass
-        return eval("".join(in_file.readlines()))
+
+    def ack(self):
+        """A successful reply to a Santiago request.
+
+        The response must include:
+
+        - A server.
+
+        The response may include:
+
+        - The Santiago listener that received and accepted the request.
+
+        """
+        pass
+
+    def end(self):
+        """Sent by the original requester, when it receives the server's
+        response, telling the server it needs to send no more responses.
+
+        Sent to the Santiago that first received the request.
+
+        """
+        pass
+
+    def proxy(self, key, service, server, hops):
+        """Sends the request to another server."""
+        
+        # TODO pull this off, another day.
+        return (self.santiago.instance +" is not %(server)s.  proxying request. " +
+                "%(key)s is requesting the %(service)s from %(server)s. " +
+                "%(hops)d hops remain.") % locals()
 
 
 if __name__ == "__main__":
-    santiago = SantiagoListener("fbx")
+    import cherrypy
+    import sys
+    sys.path.append(".")
+    from protocols.http import SantiagoHttpListener, SantiagoHttpSender
 
-    cherrypy.quickstart(santiago)
+    # build the Santiago
+    santiago = Santiago("nick")
+    http_listener = SantiagoHttpListener(santiago)
+    http_sender = SantiagoHttpSender(santiago)
+    santiago.add_listener(http_listener)
+    santiago.add_sender(http_sender)
+
+    # TODO move this into the table loading.
+    santiago.provide_at_location("wiki", "192.168.0.13")
+    santiago.provide_for_key("wiki", "james")
+    santiago.max_hops = 3
+    santiago.proxy_list = ("tor")
+
+    cherrypy.quickstart(http_listener)