syft.core.workers package¶
Submodules¶
syft.core.workers.base module¶
-
class
syft.core.workers.base.
BaseWorker
(hook=None, id=0, is_client_worker=False, objects={}, tmp_objects={}, known_workers={}, verbose=True, queue_size=0)¶ Bases:
abc.ABC
The BaseWorker class establishes a consistent interface for communicating between different machines about tensors, variables, models, and other network related information. It defines base functionality for storing objects within the worker (or in some cases specifically not storing them), and for keeping track of known workers across a network.
This class does NOT include any communication specific functionality such as peer-to-peer protocols, routing, node discovery, or even socket architecture. Only classes that extend the BaseWorker should carry such functionality.
Parameters: - hook (
hooks.BaseHook
) This is a reference to the hook object which overloaded the underlying deep learning framework. - id (int or string, optional) the integer or string identifier for this node
- is_client_worker (bool, optional) a boolean which determines whether this worker is associeted with an end user client. If so, it assumes that the client will maintain control over when tensors/variables/models are instantiated or deleted as opposed to handling tensor/variable/model lifecycle internally.
- objects (list of tensors, variables, or models, optional) When the worker is NOT a client worker, it stores all tensors it receives or creates in this dictionary. The key to each object is it's id.
- tmp_objects (list of tensors, variables, or models, optional) When the worker IS a client worker, it stores some tensors temporarily in this _tmp_objects simply to ensure that they do not get deallocated by the Python garbage collector while in the process of being registered. This dictionary can be emptied using the clear_tmp_objects method.
- known_workers (list of **:class:`BaseWorker` ** objects, optional) This dictionary can include all known workers on a network. Some extensions of BaseWorker will use this list to bootstrap the network.
- verbose (bool, optional) A flag for whether or not to print events to stdout.
-
add_worker
(worker) → None¶ This method adds a worker to the list of _known_workers internal to the BaseWorker. It endows this class with the ability to communicate with the remote worker being added, such as sending and receiving objects, commands, or information about the network.
Parameters: - worker (
BaseWorker
) This is an object pointer to a remote worker, which must have a unique id.
Example: >>> from syft.core.hooks import TorchHook >>> from syft.core.hooks import torch >>> from syft.core.workers import VirtualWorker >>> hook = TorchHook() Hooking into Torch... Overloading complete. >>> local = hook.local_worker >>> remote = VirtualWorker(id=1, hook=hook) >>> local.add_worker(remote) >>> x = torch.FloatTensor([1,2,3,4,5]) >>> x 1 2 3 4 5 [torch.FloatTensor of size 5] >>> x.send(remote) >>> x [torch.FloatTensor - Locations:[<VirtualWorker at 0x11848bda0>]] >>> x.get() >>> x 1 2 3 4 5 [torch.FloatTensor of size 5]
- worker (
-
compile_composite_message
()¶ Returns a composite message in a dictionary from the message queue. Eventually will take a recipient id.
- out (dict) dictionary containing the message queue compiled
as a composite message
-
compile_result
(result, owners)¶ Converts the result to a JSON serializable message for sending over PubSub.
-
get_obj
(remote_key) → a torch object¶ This method fetches a tensor from the worker's internal registry of tensors using its id. Note that this will not work on client worker because they do not store objects internally. However, it can be used on remote workers, including remote
VirtualWorker
objects, as pictured below in the example.Parameters: - remote_key (string or int) This is the id of the object to be returned.
Example: >>> from syft.core.hooks import TorchHook >>> from syft.core.hooks import torch >>> from syft.core.workers import VirtualWorker >>> hook = TorchHook() >>> local = hook.local_worker >>> remote = VirtualWorker(hook=hook, id=1) >>> local.add_worker(remote) Hooking into Torch... Overloading complete. >>> x = torch.FloatTensor([1,2,3,4,5]).send(remote) >>> x [torch.FloatTensor - Locations:[<VirtualWorker at 0x113f58c50>]] >>> x.id<PHONE_NUMBER> >>> remote.get_obj(x.id) [torch.FloatTensor - Locations:[1]]
-
get_worker
(self, id_or_worker) → BaseWorker¶ If you pass in an ID, it will attempt to find the worker object (pointer) withins self._known_workers. If you instead pass in a pointer itself, it will save that as a known_worker if it does not exist as one. This method is primarily useful becuase often tensors have to store only the ID to a foreign worker which may or may not be known by the worker that is deserializing it at the time of deserialization. This method allows for resolution of worker ids to workers to happen automatically while also making the current worker aware of new ones when discovered through other processes.
Parameters: - id_or_worker (string or int or
BaseWorker
) This is either the id of the object to be returned or the object itself.
Example: >>> from syft.core.hooks import TorchHook >>> from syft.core.hooks import torch >>> from syft.core.workers import VirtualWorker >>> # hook torch and create a local worker >>> hook = TorchHook() >>> local = hook.local_worker Hooking into Torch... Overloading complete. >>> # lets create a remote worker >>> remote = VirtualWorker(hook=hook,id=1) >>> local.add_worker(remote) >>> remote <syft.core.workers.VirtualWorker at 0x10e3a2a90> >>> # we can get the worker using it's id (1) >>> local.get_worker(1) <syft.core.workers.VirtualWorker at 0x10e3a2a90> >>> # or we can get the worker by passing in the worker >>> local.get_worker(remote) <syft.core.workers.VirtualWorker at 0x10e3a2a90>
- id_or_worker (string or int or
-
handle_command
(message)¶ Main function that handles incoming torch commands.
-
handle_register
(torch_object, obj_msg, force_attach_to_worker=False, temporary=False)¶ This function is responsible for re-registering an object when it has been previously registered with the wrong id.
Parameters: - torch_object (torch.Tensor or torch.autograd.Variable) the object
to be re-registered.
- obj_msg (dict)** the message containing the proper id.
- force_attach_to_worker (bool) if set to True, it will force the object
- to be stored in the worker's permanent registry
- even if the worker is a client worker.
- temporary (bool) If set to True, it will store the object in the
worker's temporary registry.
- out (torch.Tensor or torch.autograd.Variable) returns the object
newly registered.
-
prepare_send_object
(obj, delete_local=True, send_pointer=False)¶
-
process_command
(self, command_msg) -> (command output, list of owners)¶ Process a command message from a client worker. Returns the result of the computation and a list of the result's owners.
Parameters: - command_msg (dict) The dictionary containing a command from another worker.
- out (command output, list of
BaseWorker
ids/objects ) This executes the command and returns its output along with a list of the owners of the tensors involved.
-
process_message_type
(message_wrapper)¶ This method takes a message wrapper and attempts to process it against known processing methods. If the method is a composite message, it unroles applies recursively
- message_wrapper (dict) Dictionary containing the message and meta information
- out (object) the response. This can be a variety
of object types. However, the object is typically only
used during testing or local development with
VirtualWorker
workers.
-
process_response
(response) → dict¶ Processes a worker's response from a command, converting it from the raw form sent over the wire into python objects leading to the execution of a command such as storing a tensor or manipulating it in some way.
Parameters: - response(string) This is the raw message received from a foreign worker or client.
- out (dict) the result of the parsing.
-
receive_msg
(message_wrapper_json, is_binary=True)¶ Receives an message from a worker and then executes its contents appropriately. The message is encoded as a binary blob.
- message (binary) the message being sent
- out (object) the response. This can be a variety
of object types. However, the object is typically only used during testing or
local development with
VirtualWorker
workers.
-
receive_obj
(self, message) -> (a torch.autograd.Variable or torch.Tensor object)¶ Functionality that receives a Tensor or Variable from another VirtualWorker (as a string), deserializes it, and registers it within the local permanent registry.
Parameters: - message(JSON string) the message encoding the object being received.
-
register_object
(obj, force_attach_to_worker=False, temporary=False, **kwargs)¶ Registers an object with the current worker node. Selects an id for the object, assigns a list of owners, and establishes whether it's a pointer or not. This method is generally not used by the client and is instead used by interal processes (hooks and workers).
Parameters: - obj (a torch.Tensor or torch.autograd.Variable) a Torch instance, e.g. Tensor or Variable to be registered
- force_attach_to_worker (bool) if set to True, it will force the object to be stored in the worker's permanent registry
- temporary (bool) If set to True, it will store the object in the worker's temporary registry.
Kwargs: - id (int or string) random integer between 0 and 1e10 or string uniquely identifying the object.
- owners (list of ** :class:`BaseWorker` objects ** or ids) owner(s) of the object
- is_pointer (bool, optional) Whether or not the tensor being registered contains the data locally or is instead a pointer to a tensor that lives on a different worker.
-
request_obj
(self, obj_id, sender)¶ This method requests that another VirtualWorker send an object to the local one. In the case that the local one is a client, it simply returns the object. In the case that the local worker is not a client, it stores the object in the permanent registry.
Parameters: - obj_id (str or int) the id of the object being requested
- sender (
VirtualWorker
) the worker who currently has the object who is being requested to send it.
-
rm_obj
(remote_key)¶ This method removes an object from the permament object registory if it exists.
Parameters: - remote_key(int or string) the id of the object to be removed
-
send_msg
(message, message_type, recipient)¶ Sends a string message to another worker with message_type information indicating how the message should be processed.
Parameters: - recipient (
VirtualWorker
) the worker being sent a message. - message (string) the message being sent
- message_type (string) the type of message being sent. This affects how
the message is processed by the recipient. The types of message are described
in
receive_msg()
. - out (object) the response from the message being sent. This can be a variety
of object types. However, the object is typically only used during testing or
local development with
VirtualWorker
workers.
- recipient (
-
send_obj
(self, obj, recipient, delete_local=True) → obj¶ Sends an object to another
VirtualWorker
and, by default, removes it from the local worker. It also returns the object as a special case when the caller is a client. In most cases, send_obj would be handled on the other side by storing it in the permament registry. However, for VirtualWorkers attached to clients, we don't want this to occur. Thus, this method returns the object as a workaround. SeeVirtualWorker.request_obj()
for more deatils.Parameters: - obj (object) a python object to be sent
- recipient (
VirtualWorker
) the worker object to send the message to. - delete_local (bool, optional) when set to true, it deletes the version of the
object in the local registry.
-
send_torch_command
(self, recipient, message, response_handler, timeout=10) → object¶ This method sends a message to another worker in a way that hangs... waiting until the worker responds with a message. It then processes the response using a response handler
Parameters: - recipient (
VirtualWorker
) the worker being sent a message. - message (string) the message being sent
- recipient (
-
set_obj
(remote_key, value, force=False, tmp=False)¶ This method adds an object (such as a tensor) to one of the internal object registries. The main registry holds objects for a long period of time (until they are explicitly deleted) while the temporary registory will delete all object references contained therein when the cleanup method is called (self._clear_tmp_objects).
The reason we have a temporary registry for clijen workers (instead of putting everything in the permanent one) is because when the client creates a reference to an object (x = torch.zeros(10)), we want to ensure that the client contains the only pointer to that object so that when the client deletes the object that __del__ gets called. If, however, the object was also being stored in the permanent registry (self._objects), then when the variable went out of scope on the client, the python garbage collector wouldn't call __del__ because the internal registry would still have a reference to it. So, we use the temporary registry during the construction of the object but then delete all references to the object (other than the client's) once object construction and (recursive) registration is complete.
When the worker is not a client worker (self._is_client_worker==False), this method just saves an object into the permanent registry, where it remains until it is explicitly deleted using self._rm_obj.
Parameters: - remote_key (int) This is an object id to an object to be stored in memory.
- **value (torch.Tensor or torch.autograd.Variable) ** the object to be stored in memory.
- force (bool, optional) if set to True, this will force the object to be stored in permenent memory, even if the current worker is a client worker (Default: False).
- tmp (bobol, optional) if set to True, this will allow an object to be stored in temporary memory if and only if the worker is also a client worker. If set to false, the object will not be stored in temporary memory, even if the worker is a client worker.
Example: >>> from syft.core.hooks import TorchHook >>> from syft.core.hooks import torch >>> from syft.core.workers import VirtualWorker >>> import json >>> # >>> # hook pytorch and get worker >>> hook = TorchHook() >>> local = hook.local_worker Hooking into Torch... Overloading complete. >>> # showing that the current worker is a client worker (default) >>> local.is_client_worker True >>> message_obj = json.loads(' {"torch_type": "torch.FloatTensor", "data": [1.0, 2.0, 3.0, 4.0, 5.0], "id":<PHONE_NUMBER>, "owners": [1], "is_pointer": false}') >>> obj_type = hook.guard.types_guard(message_obj['torch_type']) >>> unregistered_tensor = torch.FloatTensor.deser(obj_type,message_obj) >>> unregistered_tensor 1 2 3 4 5 [torch.FloatTensor of size 5] >>> # calling set_obj on a client worker when force=False (default) >>> local.set_obj(unregistered_tensor.id, unregistered_tensor) >>> local._objects {} >>> # calling set_obj on a client worker when force=True >>> local.set_obj(unregistered_tensor.id, unregistered_tensor, force=True) >>> local._objects {3070459025: 1 2 3 4 5 [torch.FloatTensor of size 5]}
-
whoami
()¶ Returns metadata information about the worker. This function returns the default which is the id and type of the current worker. Other worker types can extend this function with additional metadata such as network information.
- hook (
syft.core.workers.socket module¶
-
class
syft.core.workers.socket.
SocketWorker
(hook=None, hostname='localhost', port=8110, max_connections=5, id=0, is_client_worker=True, objects={}, tmp_objects={}, known_workers={}, verbose=True, is_pointer=False, queue_size=0)¶ Bases:
syft.core.workers.base.BaseWorker
A worker capable of performing the functions of a BaseWorker across a socket connection. Note that the worker is NOT responsible for creating the socket connection.
Parameters: - hook (
hooks.BaseHook
) This is a reference to the hook object which overloaded the underlying deep learning framework. - id (int or string, optional) the integer or string identifier for this node
- is_client_worker (bool, optional) a boolean which determines whether this worker is associeted with an end user client. If so, it assumes that the client will maintain control over when tensors/variables/models are instantiated or deleted as opposed to handling tensor/variable/model lifecycle internally.
- objects (list of tensors, variables, or models, optional) When the worker is NOT a client worker, it stores all tensors it receives or creates in this dictionary. The key to each object is it's id.
- tmp_objects (list of tensors, variables, or models, optional) When the worker IS a client worker, it stores some tensors temporarily in this _tmp_objects simply to ensure that they do not get deallocated by the Python garbage collector while in the process of being registered. This dictionary can be emptied using the clear_tmp_objects method.
- known_workers (list of **:class:`BaseWorker` ** objects, optional) This dictionary can include all known workers.
- verbose (bool, optional) A flag for whether or not to print events to stdout.
Example Server: >>> from syft.core.hooks import TorchHook >>> from syft.core.workers import VirtualWorker, SocketWorker >>> hook = TorchHook() Hooking into Torch... Overloading complete. >>> local_worker = SocketWorker(hook=hook, id=2, port=8181, is_pointer=False, is_client_worker=False) Starting Socket Worker... Ready to receive commands...
Example Client: >>> import torch >>> from syft.core.hooks import TorchHook >>> from syft.core.workers import SocketWorker >>> hook = TorchHook(local_worker=SocketWorker(id=0, port=8182)) Starting Socket Worker... Ready! Hooking into Torch... Overloading complete. >>> remote_client = SocketWorker(hook=hook,id=2, port=8181, is_pointer=True) >>> hook.local_worker.add_worker(remote_client) Attaching Pointer to Socket Worker... >>> x = torch.FloatTensor([1,2,3,4,5]).send(remote_client) >>> x2 = torch.FloatTensor([1,2,3,4,4]).send(remote_client) >>> y = x + x2 + x >>> y [torch.FloatTensor - Locations:[<syft.core.workers.SocketWorker object at 0x7f94eaaa6630>]] >>> y.get() 3 6 9 12 14 [torch.FloatTensor of size 5]
-
listen
(num_messages=-1)¶ Starts SocketWorker server on the correct port and handles message as they are received.
-
whoami
()¶ Returns metadata information about the worker. This function returns the default which is the id and type of the current worker. Other worker types can extend this function with additional metadata such as network information.
- hook (
syft.core.workers.virtual module¶
-
class
syft.core.workers.virtual.
VirtualWorker
(hook, id=0, is_client_worker=False, objects={}, tmp_objects={}, known_workers={}, verbose=False, queue_size=0)¶ Bases:
syft.core.workers.base.BaseWorker
A virtualized worker simulating the existence of a remote machine. It is intended as a testing, development, and performance evaluation tool that exists independent of a live or local network of workers. You don't even have to be connected to the internet to create a pool of Virtual Workers and start running computations on them.
Parameters: - hook (
hooks.BaseHook
) This is a reference to the hook object which overloaded the underlying deep learning framework. - id (int or string, optional) the integer or string identifier for this node
- is_client_worker (bool, optional) a boolean which determines whether this worker is associeted with an end user client. If so, it assumes that the client will maintain control over when tensors/variables/models are instantiated or deleted as opposed to handling tensor/variable/model lifecycle internally.
- objects (list of tensors, variables, or models, optional) When the worker is NOT a client worker, it stores all tensors it receives or creates in this dictionary. The key to each object is it's id.
- tmp_objects (list of tensors, variables, or models, optional) When the worker IS a client worker, it stores some tensors temporarily in this _tmp_objects simply to ensure that they do not get deallocated by the Python garbage collector while in the process of being registered. This dictionary can be emptied using the clear_tmp_objects method.
- known_workers (list of **:class:`BaseWorker` ** objects, optional) This dictionary can include all known workers.
- verbose (bool, optional) A flag for whether or not to print events to stdout.
Example: >>> from syft.core.hooks import TorchHook >>> from syft.core.hooks import torch >>> from syft.core.workers import VirtualWorker >>> hook = TorchHook() Hooking into Torch... Overloading complete. >>> local = hook.local_worker >>> remote = VirtualWorker(id=1, hook=hook) >>> local.add_worker(remote) >>> x = torch.FloatTensor([1,2,3,4,5]) >>> x 1 2 3 4 5 [torch.FloatTensor of size 5] >>> x.send(remote) >>> x [torch.FloatTensor - Locations:[<syft.core.workers.VirtualWorker object at 0x11848bda0>]] >>> x.get() >>> x 1 2 3 4 5 [torch.FloatTensor of size 5]
- hook (
Module contents¶
Interfaces for communicating about objects between Clients and Workers
-
class
syft.core.workers.
BaseWorker
(hook=None, id=0, is_client_worker=False, objects={}, tmp_objects={}, known_workers={}, verbose=True, queue_size=0)¶ Bases:
abc.ABC
The BaseWorker class establishes a consistent interface for communicating between different machines about tensors, variables, models, and other network related information. It defines base functionality for storing objects within the worker (or in some cases specifically not storing them), and for keeping track of known workers across a network.
This class does NOT include any communication specific functionality such as peer-to-peer protocols, routing, node discovery, or even socket architecture. Only classes that extend the BaseWorker should carry such functionality.
Parameters: - hook (
hooks.BaseHook
) This is a reference to the hook object which overloaded the underlying deep learning framework. - id (int or string, optional) the integer or string identifier for this node
- is_client_worker (bool, optional) a boolean which determines whether this worker is associeted with an end user client. If so, it assumes that the client will maintain control over when tensors/variables/models are instantiated or deleted as opposed to handling tensor/variable/model lifecycle internally.
- objects (list of tensors, variables, or models, optional) When the worker is NOT a client worker, it stores all tensors it receives or creates in this dictionary. The key to each object is it's id.
- tmp_objects (list of tensors, variables, or models, optional) When the worker IS a client worker, it stores some tensors temporarily in this _tmp_objects simply to ensure that they do not get deallocated by the Python garbage collector while in the process of being registered. This dictionary can be emptied using the clear_tmp_objects method.
- known_workers (list of **:class:`BaseWorker` ** objects, optional) This dictionary can include all known workers on a network. Some extensions of BaseWorker will use this list to bootstrap the network.
- verbose (bool, optional) A flag for whether or not to print events to stdout.
-
add_worker
(worker) → None¶ This method adds a worker to the list of _known_workers internal to the BaseWorker. It endows this class with the ability to communicate with the remote worker being added, such as sending and receiving objects, commands, or information about the network.
Parameters: - worker (
BaseWorker
) This is an object pointer to a remote worker, which must have a unique id.
Example: >>> from syft.core.hooks import TorchHook >>> from syft.core.hooks import torch >>> from syft.core.workers import VirtualWorker >>> hook = TorchHook() Hooking into Torch... Overloading complete. >>> local = hook.local_worker >>> remote = VirtualWorker(id=1, hook=hook) >>> local.add_worker(remote) >>> x = torch.FloatTensor([1,2,3,4,5]) >>> x 1 2 3 4 5 [torch.FloatTensor of size 5] >>> x.send(remote) >>> x [torch.FloatTensor - Locations:[<VirtualWorker at 0x11848bda0>]] >>> x.get() >>> x 1 2 3 4 5 [torch.FloatTensor of size 5]
- worker (
-
compile_composite_message
()¶ Returns a composite message in a dictionary from the message queue. Eventually will take a recipient id.
- out (dict) dictionary containing the message queue compiled
as a composite message
-
compile_result
(result, owners)¶ Converts the result to a JSON serializable message for sending over PubSub.
-
get_obj
(remote_key) → a torch object¶ This method fetches a tensor from the worker's internal registry of tensors using its id. Note that this will not work on client worker because they do not store objects internally. However, it can be used on remote workers, including remote
VirtualWorker
objects, as pictured below in the example.Parameters: - remote_key (string or int) This is the id of the object to be returned.
Example: >>> from syft.core.hooks import TorchHook >>> from syft.core.hooks import torch >>> from syft.core.workers import VirtualWorker >>> hook = TorchHook() >>> local = hook.local_worker >>> remote = VirtualWorker(hook=hook, id=1) >>> local.add_worker(remote) Hooking into Torch... Overloading complete. >>> x = torch.FloatTensor([1,2,3,4,5]).send(remote) >>> x [torch.FloatTensor - Locations:[<VirtualWorker at 0x113f58c50>]] >>> x.id<PHONE_NUMBER> >>> remote.get_obj(x.id) [torch.FloatTensor - Locations:[1]]
-
get_worker
(self, id_or_worker) → BaseWorker¶ If you pass in an ID, it will attempt to find the worker object (pointer) withins self._known_workers. If you instead pass in a pointer itself, it will save that as a known_worker if it does not exist as one. This method is primarily useful becuase often tensors have to store only the ID to a foreign worker which may or may not be known by the worker that is deserializing it at the time of deserialization. This method allows for resolution of worker ids to workers to happen automatically while also making the current worker aware of new ones when discovered through other processes.
Parameters: - id_or_worker (string or int or
BaseWorker
) This is either the id of the object to be returned or the object itself.
Example: >>> from syft.core.hooks import TorchHook >>> from syft.core.hooks import torch >>> from syft.core.workers import VirtualWorker >>> # hook torch and create a local worker >>> hook = TorchHook() >>> local = hook.local_worker Hooking into Torch... Overloading complete. >>> # lets create a remote worker >>> remote = VirtualWorker(hook=hook,id=1) >>> local.add_worker(remote) >>> remote <syft.core.workers.VirtualWorker at 0x10e3a2a90> >>> # we can get the worker using it's id (1) >>> local.get_worker(1) <syft.core.workers.VirtualWorker at 0x10e3a2a90> >>> # or we can get the worker by passing in the worker >>> local.get_worker(remote) <syft.core.workers.VirtualWorker at 0x10e3a2a90>
- id_or_worker (string or int or
-
handle_command
(message)¶ Main function that handles incoming torch commands.
-
handle_register
(torch_object, obj_msg, force_attach_to_worker=False, temporary=False)¶ This function is responsible for re-registering an object when it has been previously registered with the wrong id.
Parameters: - torch_object (torch.Tensor or torch.autograd.Variable) the object
to be re-registered.
- obj_msg (dict)** the message containing the proper id.
- force_attach_to_worker (bool) if set to True, it will force the object
- to be stored in the worker's permanent registry
- even if the worker is a client worker.
- temporary (bool) If set to True, it will store the object in the
worker's temporary registry.
- out (torch.Tensor or torch.autograd.Variable) returns the object
newly registered.
-
prepare_send_object
(obj, delete_local=True, send_pointer=False)¶
-
process_command
(self, command_msg) -> (command output, list of owners)¶ Process a command message from a client worker. Returns the result of the computation and a list of the result's owners.
Parameters: - command_msg (dict) The dictionary containing a command from another worker.
- out (command output, list of
BaseWorker
ids/objects ) This executes the command and returns its output along with a list of the owners of the tensors involved.
-
process_message_type
(message_wrapper)¶ This method takes a message wrapper and attempts to process it against known processing methods. If the method is a composite message, it unroles applies recursively
- message_wrapper (dict) Dictionary containing the message and meta information
- out (object) the response. This can be a variety
of object types. However, the object is typically only
used during testing or local development with
VirtualWorker
workers.
-
process_response
(response) → dict¶ Processes a worker's response from a command, converting it from the raw form sent over the wire into python objects leading to the execution of a command such as storing a tensor or manipulating it in some way.
Parameters: - response(string) This is the raw message received from a foreign worker or client.
- out (dict) the result of the parsing.
-
receive_msg
(message_wrapper_json, is_binary=True)¶ Receives an message from a worker and then executes its contents appropriately. The message is encoded as a binary blob.
- message (binary) the message being sent
- out (object) the response. This can be a variety
of object types. However, the object is typically only used during testing or
local development with
VirtualWorker
workers.
-
receive_obj
(self, message) -> (a torch.autograd.Variable or torch.Tensor object)¶ Functionality that receives a Tensor or Variable from another VirtualWorker (as a string), deserializes it, and registers it within the local permanent registry.
Parameters: - message(JSON string) the message encoding the object being received.
-
register_object
(obj, force_attach_to_worker=False, temporary=False, **kwargs)¶ Registers an object with the current worker node. Selects an id for the object, assigns a list of owners, and establishes whether it's a pointer or not. This method is generally not used by the client and is instead used by interal processes (hooks and workers).
Parameters: - obj (a torch.Tensor or torch.autograd.Variable) a Torch instance, e.g. Tensor or Variable to be registered
- force_attach_to_worker (bool) if set to True, it will force the object to be stored in the worker's permanent registry
- temporary (bool) If set to True, it will store the object in the worker's temporary registry.
Kwargs: - id (int or string) random integer between 0 and 1e10 or string uniquely identifying the object.
- owners (list of ** :class:`BaseWorker` objects ** or ids) owner(s) of the object
- is_pointer (bool, optional) Whether or not the tensor being registered contains the data locally or is instead a pointer to a tensor that lives on a different worker.
-
request_obj
(self, obj_id, sender)¶ This method requests that another VirtualWorker send an object to the local one. In the case that the local one is a client, it simply returns the object. In the case that the local worker is not a client, it stores the object in the permanent registry.
Parameters: - obj_id (str or int) the id of the object being requested
- sender (
VirtualWorker
) the worker who currently has the object who is being requested to send it.
-
rm_obj
(remote_key)¶ This method removes an object from the permament object registory if it exists.
Parameters: - remote_key(int or string) the id of the object to be removed
-
send_msg
(message, message_type, recipient)¶ Sends a string message to another worker with message_type information indicating how the message should be processed.
Parameters: - recipient (
VirtualWorker
) the worker being sent a message. - message (string) the message being sent
- message_type (string) the type of message being sent. This affects how
the message is processed by the recipient. The types of message are described
in
receive_msg()
. - out (object) the response from the message being sent. This can be a variety
of object types. However, the object is typically only used during testing or
local development with
VirtualWorker
workers.
- recipient (
-
send_obj
(self, obj, recipient, delete_local=True) → obj¶ Sends an object to another
VirtualWorker
and, by default, removes it from the local worker. It also returns the object as a special case when the caller is a client. In most cases, send_obj would be handled on the other side by storing it in the permament registry. However, for VirtualWorkers attached to clients, we don't want this to occur. Thus, this method returns the object as a workaround. SeeVirtualWorker.request_obj()
for more deatils.Parameters: - obj (object) a python object to be sent
- recipient (
VirtualWorker
) the worker object to send the message to. - delete_local (bool, optional) when set to true, it deletes the version of the
object in the local registry.
-
send_torch_command
(self, recipient, message, response_handler, timeout=10) → object¶ This method sends a message to another worker in a way that hangs... waiting until the worker responds with a message. It then processes the response using a response handler
Parameters: - recipient (
VirtualWorker
) the worker being sent a message. - message (string) the message being sent
- recipient (
-
set_obj
(remote_key, value, force=False, tmp=False)¶ This method adds an object (such as a tensor) to one of the internal object registries. The main registry holds objects for a long period of time (until they are explicitly deleted) while the temporary registory will delete all object references contained therein when the cleanup method is called (self._clear_tmp_objects).
The reason we have a temporary registry for clijen workers (instead of putting everything in the permanent one) is because when the client creates a reference to an object (x = torch.zeros(10)), we want to ensure that the client contains the only pointer to that object so that when the client deletes the object that __del__ gets called. If, however, the object was also being stored in the permanent registry (self._objects), then when the variable went out of scope on the client, the python garbage collector wouldn't call __del__ because the internal registry would still have a reference to it. So, we use the temporary registry during the construction of the object but then delete all references to the object (other than the client's) once object construction and (recursive) registration is complete.
When the worker is not a client worker (self._is_client_worker==False), this method just saves an object into the permanent registry, where it remains until it is explicitly deleted using self._rm_obj.
Parameters: - remote_key (int) This is an object id to an object to be stored in memory.
- **value (torch.Tensor or torch.autograd.Variable) ** the object to be stored in memory.
- force (bool, optional) if set to True, this will force the object to be stored in permenent memory, even if the current worker is a client worker (Default: False).
- tmp (bobol, optional) if set to True, this will allow an object to be stored in temporary memory if and only if the worker is also a client worker. If set to false, the object will not be stored in temporary memory, even if the worker is a client worker.
Example: >>> from syft.core.hooks import TorchHook >>> from syft.core.hooks import torch >>> from syft.core.workers import VirtualWorker >>> import json >>> # >>> # hook pytorch and get worker >>> hook = TorchHook() >>> local = hook.local_worker Hooking into Torch... Overloading complete. >>> # showing that the current worker is a client worker (default) >>> local.is_client_worker True >>> message_obj = json.loads(' {"torch_type": "torch.FloatTensor", "data": [1.0, 2.0, 3.0, 4.0, 5.0], "id":<PHONE_NUMBER>, "owners": [1], "is_pointer": false}') >>> obj_type = hook.guard.types_guard(message_obj['torch_type']) >>> unregistered_tensor = torch.FloatTensor.deser(obj_type,message_obj) >>> unregistered_tensor 1 2 3 4 5 [torch.FloatTensor of size 5] >>> # calling set_obj on a client worker when force=False (default) >>> local.set_obj(unregistered_tensor.id, unregistered_tensor) >>> local._objects {} >>> # calling set_obj on a client worker when force=True >>> local.set_obj(unregistered_tensor.id, unregistered_tensor, force=True) >>> local._objects {3070459025: 1 2 3 4 5 [torch.FloatTensor of size 5]}
-
whoami
()¶ Returns metadata information about the worker. This function returns the default which is the id and type of the current worker. Other worker types can extend this function with additional metadata such as network information.
- hook (
-
class
syft.core.workers.
SocketWorker
(hook=None, hostname='localhost', port=8110, max_connections=5, id=0, is_client_worker=True, objects={}, tmp_objects={}, known_workers={}, verbose=True, is_pointer=False, queue_size=0)¶ Bases:
syft.core.workers.base.BaseWorker
A worker capable of performing the functions of a BaseWorker across a socket connection. Note that the worker is NOT responsible for creating the socket connection.
Parameters: - hook (
hooks.BaseHook
) This is a reference to the hook object which overloaded the underlying deep learning framework. - id (int or string, optional) the integer or string identifier for this node
- is_client_worker (bool, optional) a boolean which determines whether this worker is associeted with an end user client. If so, it assumes that the client will maintain control over when tensors/variables/models are instantiated or deleted as opposed to handling tensor/variable/model lifecycle internally.
- objects (list of tensors, variables, or models, optional) When the worker is NOT a client worker, it stores all tensors it receives or creates in this dictionary. The key to each object is it's id.
- tmp_objects (list of tensors, variables, or models, optional) When the worker IS a client worker, it stores some tensors temporarily in this _tmp_objects simply to ensure that they do not get deallocated by the Python garbage collector while in the process of being registered. This dictionary can be emptied using the clear_tmp_objects method.
- known_workers (list of **:class:`BaseWorker` ** objects, optional) This dictionary can include all known workers.
- verbose (bool, optional) A flag for whether or not to print events to stdout.
Example Server: >>> from syft.core.hooks import TorchHook >>> from syft.core.workers import VirtualWorker, SocketWorker >>> hook = TorchHook() Hooking into Torch... Overloading complete. >>> local_worker = SocketWorker(hook=hook, id=2, port=8181, is_pointer=False, is_client_worker=False) Starting Socket Worker... Ready to receive commands...
Example Client: >>> import torch >>> from syft.core.hooks import TorchHook >>> from syft.core.workers import SocketWorker >>> hook = TorchHook(local_worker=SocketWorker(id=0, port=8182)) Starting Socket Worker... Ready! Hooking into Torch... Overloading complete. >>> remote_client = SocketWorker(hook=hook,id=2, port=8181, is_pointer=True) >>> hook.local_worker.add_worker(remote_client) Attaching Pointer to Socket Worker... >>> x = torch.FloatTensor([1,2,3,4,5]).send(remote_client) >>> x2 = torch.FloatTensor([1,2,3,4,4]).send(remote_client) >>> y = x + x2 + x >>> y [torch.FloatTensor - Locations:[<syft.core.workers.SocketWorker object at 0x7f94eaaa6630>]] >>> y.get() 3 6 9 12 14 [torch.FloatTensor of size 5]
-
listen
(num_messages=-1)¶ Starts SocketWorker server on the correct port and handles message as they are received.
-
whoami
()¶ Returns metadata information about the worker. This function returns the default which is the id and type of the current worker. Other worker types can extend this function with additional metadata such as network information.
- hook (
-
class
syft.core.workers.
VirtualWorker
(hook, id=0, is_client_worker=False, objects={}, tmp_objects={}, known_workers={}, verbose=False, queue_size=0)¶ Bases:
syft.core.workers.base.BaseWorker
A virtualized worker simulating the existence of a remote machine. It is intended as a testing, development, and performance evaluation tool that exists independent of a live or local network of workers. You don't even have to be connected to the internet to create a pool of Virtual Workers and start running computations on them.
Parameters: - hook (
hooks.BaseHook
) This is a reference to the hook object which overloaded the underlying deep learning framework. - id (int or string, optional) the integer or string identifier for this node
- is_client_worker (bool, optional) a boolean which determines whether this worker is associeted with an end user client. If so, it assumes that the client will maintain control over when tensors/variables/models are instantiated or deleted as opposed to handling tensor/variable/model lifecycle internally.
- objects (list of tensors, variables, or models, optional) When the worker is NOT a client worker, it stores all tensors it receives or creates in this dictionary. The key to each object is it's id.
- tmp_objects (list of tensors, variables, or models, optional) When the worker IS a client worker, it stores some tensors temporarily in this _tmp_objects simply to ensure that they do not get deallocated by the Python garbage collector while in the process of being registered. This dictionary can be emptied using the clear_tmp_objects method.
- known_workers (list of **:class:`BaseWorker` ** objects, optional) This dictionary can include all known workers.
- verbose (bool, optional) A flag for whether or not to print events to stdout.
Example: >>> from syft.core.hooks import TorchHook >>> from syft.core.hooks import torch >>> from syft.core.workers import VirtualWorker >>> hook = TorchHook() Hooking into Torch... Overloading complete. >>> local = hook.local_worker >>> remote = VirtualWorker(id=1, hook=hook) >>> local.add_worker(remote) >>> x = torch.FloatTensor([1,2,3,4,5]) >>> x 1 2 3 4 5 [torch.FloatTensor of size 5] >>> x.send(remote) >>> x [torch.FloatTensor - Locations:[<syft.core.workers.VirtualWorker object at 0x11848bda0>]] >>> x.get() >>> x 1 2 3 4 5 [torch.FloatTensor of size 5]
- hook (
-
class
syft.core.workers.
WebSocketWorker
(hook=None, hostname='localhost', port=8110, max_connections=5, id=0, is_client_worker=True, objects={}, tmp_objects={}, known_workers={}, verbose=True, is_pointer=False, queue_size=0)¶ Bases:
syft.core.workers.base.BaseWorker
A worker capable of performing the functions of a BaseWorker across a websocket connection. This worker essentially can replace Socket Worker.
Parameters: - hook (
hooks.BaseHook
) This is a reference to the hook object which overloaded the underlying deep learning framework. - id (int or string, optional) the integer or string identifier for this node
- is_client_worker (bool, optional) a boolean which determines whether this worker is associated with an end user client. If so, it assumes that the client will maintain control over when tensors/variables/models are instantiated or deleted as opposed to handling tensor/variable/model lifecycle internally.
- objects (list of tensors, variables, or models, optional) When the worker is NOT a client worker, it stores all tensors it receives or creates in this dictionary. The key to each object is it's id.
- tmp_objects (list of tensors, variables, or models, optional) When the worker IS a client worker, it stores some tensors temporarily in this _tmp_objects simply to ensure that they do not get deallocated by the Python garbage collector while in the process of being registered. This dictionary can be emptied using the clear_tmp_objects method.
- known_workers (list of **:class:`BaseWorker` ** objects, optional) This dictionary can include all known workers.
- verbose (bool, optional) A flag for whether or not to print events to stdout.
Example Server: >>> from syft.core.hooks import TorchHook >>> from syft.core.workers import WebSocketWorker >>> hook = TorchHook() Hooking into Torch... Overloading complete. >>> local_worker = WebSocketWorker(hook=hook, id=2, port=8181, is_pointer=False, is_client_worker=False) Starting a Websocket Worker.... Ready to receive commands.... Server Socket has been initialized
Example Client: >>> import torch >>> from syft.core.hooks import TorchHook >>> from syft.core.workers import WebSocketWorker >>> hook = TorchHook(local_worker=WebSocketWorker(id=0, port=8182)) Starting Socket Worker... Ready! Hooking into Torch... Overloading complete. >>> remote_client = WebSocketWorker(hook=hook,id=2, port=8181, is_pointer=True) >>> hook.local_worker.add_worker(remote_client) Attaching Pointer to Socket Worker... >>> x = torch.FloatTensor([1,2,3,4,5]).send(remote_client) >>> x2 = torch.FloatTensor([1,2,3,4,4]).send(remote_client) >>> y = x + x2 + x >>> y [torch.FloatTensor - Locations:[<syft.core.workers.SocketWorker object at 0x7f94eaaa6630>]] >>> y.get() 3 6 9 12 14 [torch.FloatTensor of size 5]
-
send_msg
(message, message_type, recipient)¶ Sends a string message to another worker with message_type information indicating how the message should be processed.
Parameters: - recipient (
VirtualWorker
) the worker being sent a message. - message (string) the message being sent
- message_type (string) the type of message being sent. This affects how
the message is processed by the recipient. The types of message are described
in
receive_msg()
. - out (object) the response from the message being sent. This can be a variety
of object types. However, the object is typically only used during testing or
local development with
VirtualWorker
workers.
- recipient (
-
whoami
()¶ Returns metadata information about the worker. This method returns the default which is the id and uri of the worker.
- hook (


Astronomical Image of day
Images from Mars Rovers
Earth Natural Event Tracker
Asronomical Picture of day


The world is waiting for you, with open arms
What destination you choose,
Travel Travis
is the right booking agency.


Provides functions and structures to be use for session recording.
More...
Go to the source code of this file.
|
| struct | guac_recording |
| | An in-progress session recording, attached to a guac_client instance such that output Guacamole instructions may be dynamically intercepted and written to a file. More...
|
| |
|
| guac_recording * | guac_recording_create (guac_client *client, const char *path, const char *name, int create_path, int include_output, int include_mouse, int include_touch, int include_keys) |
| | Replaces the socket of the given client such that all further Guacamole protocol output will be copied into a file within the given path and having the given name. More...
|
| |
| void | guac_recording_free (guac_recording *recording) |
| | Frees the resources associated with the given in-progress recording. More...
|
| |
| void | guac_recording_report_mouse (guac_recording *recording, int x, int y, int button_mask) |
| | Reports the current mouse position and button state within the recording. More...
|
| |
| void | guac_recording_report_touch (guac_recording *recording, int id, int x, int y, int x_radius, int y_radius, double angle, double force) |
| | Reports the current state of a touch contact within the recording. More...
|
| |
| void | guac_recording_report_key (guac_recording *recording, int keysym, int pressed) |
| | Reports a change in the state of an individual key within the recording. More...
|
| |
Provides functions and structures to be use for session recording.
◆ GUAC_COMMON_RECORDING_MAX_SUFFIX
| #define GUAC_COMMON_RECORDING_MAX_SUFFIX 255 |
The maximum numeric value allowed for the .1, .2, .3, etc.
suffix appended to the end of the session recording filename if a recording having the requested name already exists.
◆ guac_recording_create()
| guac_recording* guac_recording_create |
( |
guac_client * |
client, |
|
|
const char * |
path, |
|
|
const char * |
name, |
|
|
int |
create_path, |
|
|
int |
include_output, |
|
|
int |
include_mouse, |
|
|
int |
include_touch, |
|
|
int |
include_keys |
|
) |
| |
Replaces the socket of the given client such that all further Guacamole protocol output will be copied into a file within the given path and having the given name.
If the create_path flag is non-zero, the given path will be created if it does not yet exist. If creation of the recording file or path fails, error messages will automatically be logged, and no recording will be written. The recording will automatically be closed once the client is freed.
- Parameters
-
| client | The client whose output should be copied to a recording file. |
| path | The full absolute path to a directory in which the recording file should be created. |
| name | The base name to use for the recording file created within the specified path. |
| create_path | Zero if the specified path MUST exist for the recording file to be written, or non-zero if the path should be created if it does not yet exist. |
| include_output | Non-zero if output which is broadcast to each connected client (graphics, streams, etc.) should be included in the session recording, zero otherwise. Including output is necessary for any recording which must later be viewable as video. |
| include_mouse | Non-zero if changes to mouse state, such as position and buttons pressed or released, should be included in the session recording, zero otherwise. Including mouse state is necessary for the mouse cursor to be rendered in any resulting video. |
| include_touch | Non-zero if touch events should be included in the session recording, zero otherwise. Depending on whether the remote desktop will automatically provide graphical feedback for touches, including touch events may be necessary for multi-touch interactions to be rendered in any resulting video. |
| include_keys | Non-zero if keys pressed and released should be included in the session recording, zero otherwise. Including key events within the recording may be necessary in certain auditing contexts, but should only be done with caution. Key events can easily contain sensitive information, such as passwords, credit card numbers, etc. |
- Returns
- A new guac_recording structure representing the in-progress recording if the recording file has been successfully created and a recording will be written, NULL otherwise.
◆ guac_recording_free()
Frees the resources associated with the given in-progress recording.
Note that, due to the manner that recordings are attached to the guac_client, the underlying guac_socket is not freed. The guac_socket will be automatically freed when the guac_client is freed.
- Parameters
-
◆ guac_recording_report_key()
| void guac_recording_report_key |
( |
guac_recording * |
recording, |
|
|
int |
keysym, |
|
|
int |
pressed |
|
) |
| |
Reports a change in the state of an individual key within the recording.
- Parameters
-
| recording | The guac_recording associated with the key that was pressed or released. |
| keysym | The X11 keysym of the key that was pressed or released. |
| pressed | Non-zero if the key represented by the given keysym is currently pressed, zero if it is released. |
◆ guac_recording_report_mouse()
| void guac_recording_report_mouse |
( |
guac_recording * |
recording, |
|
|
int |
x, |
|
|
int |
y, |
|
|
int |
button_mask |
|
) |
| |
Reports the current mouse position and button state within the recording.
- Parameters
-
| recording | The guac_recording associated with the mouse that has moved. |
| x | The new X coordinate of the mouse cursor, in pixels. |
| y | The new Y coordinate of the mouse cursor, in pixels. |
| button_mask | An integer value representing the current state of each button, where the Nth bit within the integer is set to 1 if and only if the Nth mouse button is currently pressed. The lowest-order bit is the left mouse button, followed by the middle button, right button, and finally the up and down buttons of the scroll wheel. |
- See also
- GUAC_CLIENT_MOUSE_LEFT
-
GUAC_CLIENT_MOUSE_MIDDLE
-
GUAC_CLIENT_MOUSE_RIGHT
-
GUAC_CLIENT_MOUSE_SCROLL_UP
-
GUAC_CLIENT_MOUSE_SCROLL_DOWN
◆ guac_recording_report_touch()
| void guac_recording_report_touch |
( |
guac_recording * |
recording, |
|
|
int |
id, |
|
|
int |
x, |
|
|
int |
y, |
|
|
int |
x_radius, |
|
|
int |
y_radius, |
|
|
double |
angle, |
|
|
double |
force |
|
) |
| |
Reports the current state of a touch contact within the recording.
- Parameters
-
| recording | The guac_recording associated with the touch contact that has changed state. |
| id | An arbitrary integer ID which uniquely identifies this contact relative to other active contacts. |
| x | The X coordinate of the center of the touch contact. |
| y | The Y coordinate of the center of the touch contact. |
| x_radius | The X radius of the ellipse covering the general area of the touch contact, in pixels. |
| y_radius | The Y radius of the ellipse covering the general area of the touch contact, in pixels. |
| angle | The rough angle of clockwise rotation of the general area of the touch contact, in degrees. |
| force | The relative force exerted by the touch contact, where 0 is no force (the touch has been lifted) and 1 is maximum force (the maximum amount of force representable by the device). |


* Please use Chrome, Firefox and/or Edge for best site experience. Internet Explorer, Safari can have issues rendering some aspects of the charts below
Thank you for considering Ann Arbor, Michigan as a potential location for your business. Although Ann Arbor is a small college town in the midwest, we often find ourselves competing with much larger cities along with other college towns around the United States. Austin is a perfect example: we get lumped together when talking about large schools (UT and UM) but it’s important to remember that Austin is eight times larger than Ann Arbor. For that reason, the data have been normalized for population where possible.
This study is not meant to be marketing material; we produce plenty of that already It’s meant to take a deep dive into a few crucial metrics that can help distinguish one competitor from another.
Note: all metrics are compared at the county level. Ann Arbor, MI is in Washtenaw county.
These are period estimates that measure where people lived when surveyed (current residence) and where they lived one year prior. The data are collected continously over a five-year period. The flow estimates resemble the annual number of movers between counties for a five-year period.
Source of data: US Census Flows MapperTo be considered an important hub, the Ann Arbor region must be attractive to outside talent. Net population movement, both inter as well as intrastate, can potentially indicate the attractiveness of a region to outside talent, especially when viewed as proportional to the population.
In terms of net migration, Washtenaw county (Ann Arbor region) sees a higher net inflow proportional to its population that most competitor regions.
The percentage of the total employed population of a region employed in driving industries
Economies grow and prosper by their ability to make products and deliver services to people and businesses outside their geographic regions i.e. by exporting. Driving industry jobs create and support jobs in other local industries, and propel economic growth.
This ratio measures affordability by dividing the median home price by the median income
Housing cost is a key factor influencing quality of life, which affects a region's ability to attract and retain talent. Housing affordability is also a measure of inequality and access to opportunity; if the ratio is high, it can indicate a higly segregated real estate market, and a high level of income inequality. Conversely, it is also an indicator of attractiveness of a housing market.
The national unemployment rate reflects the number of unemployed people as a percentage of labor force. The labor force participation rate (LFP for short) measures the number of people in the labor force as a percentage of civilian non-institutionalized population 16 years old and over. In other words, it is the percentage of the population either working or actively seeking work.
A low unemployment rate and high labor force participation rate are signs of a well-oiled, competitive labor market. In 2018, Ann Arbor had low unemployment but also low labor force participation compared to its peers, which makes Ann Arbor an interesting place for new jobs or policy solutions.
While the previous visualization showed a snapshot of 2018, we can see the trends over the past few years for each labor metric below.
This visualization was created in the summer 2020 term for UC Berkeley Data Science course: w209 Data Visualization.
All of the code for this visualization can be found in the Github Repository.
You can find additional information around the intended users, tasks, and data for each visualization in the repository's README.


@* * This template takes a single argument, a String containing a * message to display. *@ @(questions: Seq[Question]) @* * Call the `main` template with two arguments. The first * argument is a `String` with the title of the page, the second * argument is an `Html` object containing the body of the page. *@ @main("Questions List") {


Traffic Stats Administration¶
Traffic Stats consists of three seperate components: Traffic Stats, InfluxDB, and Grafana. See below for information on installing and configuring each component as well as configuring the integration between the three and Traffic Ops.
Installation¶
Installing Traffic Stats:
- See the downloads page for Traffic Control to get the latest release.
- Follow our build intructions to generate an RPM.
- Copy the RPM to your server
- perform the following command:
sudo rpm -ivh <traffic_stats rpm>
Installing InfluxDB:
As of Traffic Stats 1.8.0, InfluxDb 1.0.0 or higher is required. For InfluxDb versions less than 1.0.0 use Traffic Stats 1.7.x
In order to store traffic stats data you will need to install InfluxDB. While not required, it is recommended to use some sort of high availability option like Influx enterprise, Influxdb Relay, or another high availability option.
Installing Grafana:
Grafana is used to display Traffic Stats/InfluxDB data in Traffic Ops. Grafana is typically run on the same server as Traffic Stats but this is not a requirement. Grafana can be installed on any server that can access InfluxDB and can be accessed by Traffic Ops. Documentation on installing Grafana can be found on the Grafana website.
Configuration¶
Configuring Traffic Stats:
Traffic Stats’ configuration file can be found in /opt/traffic_stats/conf/traffic_stats.cfg. The following values need to be configured:
- toUser: The user used to connect to Traffic Ops
- toPasswd: The password to use when connecting to Traffic Ops
- toUrl: The URL of the Traffic Ops server used by Traffic Stats
- influxUser: The user to use when connecting to InfluxDB (if configured on InfluxDB, else leave default)
- influxPassword: That password to use when connecting to InfluxDB (if configured, else leave blank)
- pollingInterval: The interval at which Traffic Monitor is polled and stats are stored in InfluxDB
- statusToMon: The status of Traffic Monitor to poll (poll ONLINE or OFFLINE traffic monitors)
- seelogConfig: The absolute path of the seelong config file
- dailySummaryPollingInterval: The interval, in seconds, at which Traffic Stats checks to see if daily stats need to be computed and stored.
- cacheRetentionPolicy: The default retention policy for cache stats
- dsRetentionPolicy: The default retention policy for deliveryservice stats
- dailySummaryRetentionPolicy: The retention policy to be used for the daily stats
- influxUrls: An array of influxdb hosts for Traffic Stats to write stats to.
Configuring InfluxDB:
As mentioned above, it is recommended that InfluxDb be running in some sort of high availability configuration. There are several ways to achieve high availabilty so it is best to consult the high availability options on the InfuxDB website.
Once InfluxDB is installed and configured, databases and retention policies need to be created. Traffic Stats writes to three different databases: cache_stats, deliveryservice_stats, and daily_stats. More information about the databases and what data is stored in each can be found on the overview page.
To easily create databases, retention policies, and continuous queries, run create_ts_databases from the /opt/traffic_stats/influxdb_tools directory on your Traffic Stats server. See the InfluxDb Tools section below for more information.
Configuring Grafana:
In Traffic Ops the Health -> Graph View tab can be configured to display grafana graphs using influxDb data. In order for this to work correctly, you will need two things 1) a parameter added to traffic ops with the graph URL (more information below) and 2) the graphs created in grafana. See below for how to create some simple graphs in grafana. These instructions assume that InfluxDB has been configured and that data has been written to it. If this is not true, you will not see any graphs.
Login to grafana as an admin user http://grafana_url:3000/login
Choose Data Sources and then Add New
Enter the necessary information to configure your data source
Click on the ‘Home’ dropdown at the top of the screen and choose New at the bottom
Click on the green menu bar (with 3 lines) at the top and choose Add Panel -> Graph
Where it says ‘No Title (click here)’ click and choose edit
Choose your data source at the bottom
You can have grafana help you create a query, or you can create your own. Here is a sample query:
SELECT sum(value)*1000 FROM "monthly"."bandwidth.cdn.1min" WHERE $timeFilter GROUP BY time(60s), cdn
Once you have the graph the way you want it, click the ‘Save Dashboard’ button at the top
You should now have a new saved graph
In order for Traffic Ops users to see Grafana graphs, Grafana will need to allow anonymous access. Information on how to configure anonymous access can be found on the configuration page of the Grafana Website.
Traffic Ops uses custom dashboards to display information about individual delivery services or cache groups. In order for the custom graphs to display correctly, the traffic_ops_*.js files need to be in the
/usr/share/grafana/public/dashboards/
directory on the grafana server. If your Grafana server is the same as your Traffic Stats server the RPM install process will take care of putting the files in place. If your grafana server is different from your Traffic Stats server, you will need to manually copy the files to the correct directory.More information on custom scripted graphs can be found in the scripted dashboards section of the Grafana documentation.
Configuring Traffic Ops for Traffic Stats:
- The influxDb servers need to be added to Traffic Ops with profile = InfluxDB. Make sure to use port 8086 in the configuration.
- The traffic stats server should be added to Traffic Ops with profile = Traffic Stats.
- Parameters for which stats will be collected are added with the release, but any changes can be made via parameters that are assigned to the Traffic Stats profile.
Configuring Traffic Ops to use Grafana Dashboards
To configure Traffic Ops to use Grafana Dashboards, you need to enter the following parameters and assign them to the GLOBAL profile. This assumes you followed the above instructions to install and configure InfluxDB and Grafana. You will need to place ‘cdn-stats’,’deliveryservice-stats’, and ‘daily-summary’ with the name of your dashboards.
parameter name parameter value all_graph_url https://<grafana_url>/dashboard/db/deliveryservice-stats cachegroup_graph_url https://<grafanaHost>/dashboard/script/traffic_ops_cachegroup.js?which= deliveryservice_graph_url https://<grafanaHost>/dashboard/script/traffic_ops_devliveryservice.js?which= server_graph_url https://<grafanaHost>/dashboard/script/traffic_ops_server.js?which= visual_status_panel_1 https://<grafanaHost>/dashboard-solo/db/cdn-stats?panelId=2&fullscreen&from=now-24h&to=now-60s visual_status_panel_2 https://<grafanaHost>/dashboard-solo/db/cdn-stats?panelId=1&fullscreen&from=now-24h&to=now-60s daily_bw_url https://<grafanaHost>/dashboard-solo/db/daily-summary?panelId=1&fullscreen&from=now-3y&to=now daily_served_url https://<grafanaHost>/dashboard-solo/db/daily-summary?panelId=2&fullscreen&from=now-3y&to=now
InfluxDb Tools¶
Under the Traffic Stats source directory there is a directory called influxdb_tools. These tools are meant to be used as one-off scripts to help a user quickly get new databases and continuous queries setup in influxdb. They are specific for traffic stats and are not meant to be generic to influxdb. Below is an brief description of each script along with how to use it.
- create/create_ts_databases.go
This script creates all databases, retention policies, and continuous queries required by traffic stats.
How to use create_ts_databases:
Pre-Requisites:
- Go 1.7 or later
- configured $GOPATH (e.g. export GOPATH=~/go)
Using create_ts_databases.go
- go to the traffic_stats/influxdb_tools/create directory
- build it by running
go build create_ts_databases.go
or simplygo build
- Run it:
./create_ts_databases -help
or./create -help
- optional flags:
- url - The influxdb url and port
- replication - The number of nodes in the cluster
- user - The user to use
- password - The password to use
- example:
./create_ts_databases -url=localhost:8086 -replication=3 -user=joe -password=mysecret
or./create -url=localhost:8086 -replication=3 -user=joe -password=mysecret
- sync_ts_databases
This script is used to sync one influxdb environment to another. Only data from continuous queries is synced as it is downsampled data and much smaller in size than syncing raw data. Possible use cases are syncing from Production to Development or Syncing a new cluster once brought online.
How to use sync_ts_databases:
Pre-Requisites:
- Go 1.7 or later
- configured $GOPATH (e.g. export GOPATH=~/go)
Using sync_ts_databases.go:
go to the traffic_stats/influxdb_tools/create directory
build it by running
go build sync_ts_databases.go
or simplygo build
- Run it
./sync_ts_databases -help
or./sync -help
- required flags:
- source-url - The URL of the source database
- target-url - The URL of the target database
- -optional flags:
- database - The database to sync (default = sync all databases)
- days - Days in the past to sync (default = sync all data)
- source-user - The user of the source database
- source-pass - The password for the source database
- target-user - The user of the target database
- target-pass - The password for the target database
- example: ./sync -source-url=http://idb-01.foo.net:8086 -target-url=http://idb-01.foo.net:8086 -database=cache_stats -days=7 -source-user=admin source-pass=mysecret


[gecode-users] Operation of precede constraint on array of sets
Tim Potter
tpot at frungy.org
Fri Nov 24 22:38:35 CET 2017
Hi everyone. I hope this isn't some silly misunderstanding but I'm having trouble figuring out how the precede() constraint works when applied to SetVarArray.
Imagine I have a set of tasks with dependencies, for example 1->2, 2->3 and 3->4 to keep things simple. I've added the following constraints:
// Array of sets of integers from {1..4}
SetVarArray deps(*this, 4, IntSet::empty, IntSet(1, 4));
// Enforce pairwise disjointedness
sequence(*this, deps);
// Final task must appear
rel(*this, setunion(deps) >= IntSet(IntArgs(1, 4)));
// Task ordering constraints
precede(*this, deps, 1, 2);
precede(*this, deps, 2, 3);
precede(*this, deps, 3, 4);
Now this should produce at least a solution like {{1}, {2}, {3}, {4}}. However I get this as my first solution in the output:
{{1..4}, {}, {}, {}}
which indicates that the precede() constraint is not behaving as I think it should.
According to to the documentation the precede(home, x, s, t) version of the value precedence constraint as applied to set variables says that s is not an element of x_j and t is an element of x_j, then i < j such that s is an element of x_i and t is not an element of x_i.
I don't understand how the solution above satisfies the above unless maybe the i < j is really acting as i <= j. This seems to be the case based on the other solutions as they're sorted but that could be a side effect of the sequence() constraint.
Does anyone know what I'm doing wrong here?
Thanks,
Tim.
More information about the users
mailing list


You are here: Home > Dive Into HTML5  >
You may well ask: “How can I start using HTML5 if older browsers don’t support it?” But the question itself is misleading.  So you can’t detect “HTML5 support,” because that doesn’t make any sense. But you can detect support for individual features, like canvas, video, or geolocation.
You may think of HTML as tags and angle brackets. That’s an important part of it, but it’s not the whole story. The HTML5 specification also defines how those angle brackets interact with JavaScript, through the Document Object Model (DOM). HTML5 doesn’t just define a video tag; there is also a corresponding DOM API for video objects in the DOM. You can use this API to detect support for different video formats, play a video, pause, mute audio, track how much of the video has been downloaded, and everything else you need to build a rich user experience around the video tag itself.
Chapter 2 and Appendix A will teach you how to properly detect support for each new HTML5 feature.
Love it or hate it, you can’t deny that HTML 4 is the most successful markup format ever. HTML5 builds on that success. You don’t need to throw away your existing markup. You don’t need to relearn things you already know. If your web application worked yesterday in HTML 4, it will still work today in HTML5. Period.
Now, if you want to improve your web applications, you’ve come to the right place. Here’s a concrete example: HTML5 supports all the form controls from HTML 4, but it also includes new input controls. Some of these are long-overdue additions like sliders and date pickers; others are more subtle. For example, the email input type looks just like a text box, but mobile browsers will customize their onscreen keyboard to make it easier to type email addresses. Older browsers that don’t support the email input type will treat it as a regular text field, and the form still works with no markup changes or scripting hacks. This means you can start improving your web forms today, even if some of your visitors are stuck on IE 6.
Read all the gory details about HTML5 forms in Chapter 9.
“Upgrading” to HTML5 can be as simple as changing your doctype. The doctype should already be on the first line of every HTML page. Previous versions of HTML defined a lot of doctypes, and choosing the right one could be tricky. In HTML5, there is only one doctype:
<!DOCTYPE html>
Upgrading to the HTML5 doctype won’t break your existing markup,
because
obsolete elements previously defined in HTML 4 will still
render in HTML5. But it will allow you to use — and validate —
new semantic elements like

,

,
, and
.
You’ll learn all about these new elements in Chapter 3.
Whether you want to draw on a canvas, play video, design better forms, or build web applications that work offline, you’ll find that HTML5 is already well-supported. Firefox, Safari, Chrome, Opera, and mobile browsers already support canvas (Chapter 4), video (Chapter 5), geolocation (Chapter 6), local storage (Chapter 7), and more. Google already supports microdata annotations (Chapter 10). Even Microsoft — rarely known for blazing the trailof standards support — supports most HTML5 features in Internet Explorer 9.
Each chapter of this book includes the all-too-familiar browser compatibility charts. But more importantly, each chapter includes a frank discussion of your options if you need to support older browsers. HTML5 features like geolocation (Chapter 6) and video (Chapter 5) were first provided by browser plugins like Gears or Flash. Other features, like canvas (Chapter 4), can be emulated entirely in JavaScript. This book will teach you how to target the native features of modern browsers, without leaving older browsers behind.
Tim Berners-Lee invented the world wide web in the early 1990s. He later founded the W3C to act as a steward of web standards, which the organization has done for more than 15 years. Here is what the W3C had to say about the future of web standards, in July 2009:
Today the Director announces that when the XHTML 2 Working Group charter expires as scheduled at the end of 2009, the charter will not be renewed. By doing so, and by increasing resources in the HTML Working Group, W3C hopes to accelerate the progress of HTML5 and clarify W3C’s position regarding the future of HTML.
HTML5 is here to stay. Let’s dive in.
|
  DID YOU KNOW? |
|
In association with Google Press, O’Reilly is distributing this book in a variety of formats, including paper, ePub, Mobi, and DRM-free PDF. The paid edition is called “HTML5: Up & Running,” and it is available now. If you liked this introduction and want to show your appreciation, you can buy “HTML5: Up & Running” with this affiliate link or buy an electronic edition directly from O’Reilly.  I do not currently accept direct donations. |
Copyright MMIX–MMXI Mark Pilgrim


Form
Enter Name
Gender
Male
Female
Email id
Enter contact No
Enter Time
Enter Your height
Attach File


The first thing to ask when dealing with Oracle database performance issues are the AWR reports. When analyzing period longer than single snapshot, you can ask for a single report but you will not get enough details. Or you can ask for separate report for each snapshot and you will have difficulties merging information from multiple files.
SPA allowes you to get multiple detailed reports and use charts to easily spot trends and hot spots, while being able to drill into details when required.
SPA is a web application that uses Node JS to serve dynamic content and MongoDB NoSQL database to store data.
To start SPA, you need following components installed:
MongoDB database.
Make sure you can connect to Mongo server using net address and port, for example:
# mongo localhost:27017
27017 is a default Mongo port number.
Node JS. Check installation with:
# node --version
v4.4.1
npm (Node Packet Manager):
# npm --version
1.4.21
Now, get SPA source code. In SPA main directory run npm to download all required dependencies with command:
# npm install
By default Mongo database called spa is created on first
application start. You can change database name (and connection details) or SPA listen port by editing
config.js
file in SPA top directory.
You are ready to start SPA with command:
# node spa.js
You should get:
SPA server up at http://localhost:3300
Point your browser (modern one, no clumsy old IE) to address printed.
These are few simple steps to get you started with using SPA for analyzing data.
Data Set is a logical unit that groups related AWR files - e.g. for the same environment and the same period.
Choose Data Set - New from top menu, enter name and description. You can change them later, each object has artificial ID assigned by Mongo database.
Choose AWR - Upload from top menu and upload AWR html reports. You may upload multiple files at once.
For best results upload files from consequentive snapshots and with the same snapshot duration. You can upload files for multiple databases / instances - they will be presented separately.
Go to previously created Data Set (choose Data Set - List from top menu and click on Data Set id).
Select AWR files tab, you will see empty list of Assigned AWR files. Click on Add AWR files to data set to see list of available AWR files. Select files and click on Add.
Click Done when finished to see list of assigned AWR files.
Go Main pane on Data Set screen and click Recalculate. This will tigger calculation of data from AWR files assigned to the Data Set. Recalculation may take few seconds, depending on the number of AWR files to go through.
Refresh the browser page when done to make sure you see updated data.
Go to Hosts pane on Data Set screen to see CPU usage for hosts captured in Data Set.
Switch to Databases pane to see charts for various database statistics. Use select drop downs to change database, instance and chart type.
Choose some chart showing SQL statements (for example SQL - Buffer Gets). Rigth click on SQL id at chart legend and choose Show SQL details from context menu.
You will get redirected to new tab with SQL text and summary of statement impact.
You can also right click area on any database chart to go details of AWR file for corresponding snapshot.
Many elements on the SPA screens are interactive and dynamic. At the same time you may use full funcionality of your browser - history navigation, multiple tabs and windows.
Charts are created using c3js library, utilizing SVG for rendering. So you need browser with SVG support (e.g. Chrome or Firefox).
Charts can be zoomed using mouse scroll and panned. Hovering mouse over chart area will display detailed values of data. Context menu on database charts allow you to navigate to AWR report for corresponding database and period.
Clicking on data item description at chart's legend hides this item from chart. It may be usefull to see data items with lower values in more details.
SQL related charts allow you to go to SQL statement details using context menu on chart legend's items.
Colors on charts are based on hashed data item names. This means that the same item will always have the same color. For example the same SQL will always get the same color on any of the charts.
Tables can be sorted by any of the column by clicking on column header. Search field filters rows containing searched text.
Range of rows can be selected with mouse when holding Shift. Multiple rows can be selected by clicking while holding Ctrl and Shift.
SQL page allows for dynamic search of SQL statements within selected Data Set and database. 2 first letters of SQL id or at least 7-letters long part of SQL text is required to trigger search. First 10 matching SQL statements are displayed.


{% extends "base.html" %} {% load static humanize i18n %}
Here's a blog tutorial in Django using the Cookiecutter-Django framework for jump starting a production-ready Django project.
Let's take this one step at a time, okay?
Learn more{{ post.overview }}


Magister Iris:
These artifacts...? Who restored them? Restoring broken artifacts to their original state is a job that requires great delicacy... Whoever worked on these is very talented.
I will send them all to the Ivory Tower. They will be precious resources for historical study. I will make sure that your name appears in the Ivory Tower scholars' research paper.
Back


No properties were found that match your criteria. Only matching property names are returned.
Reset filter


FastBin
Verification Pending
>
Thank you for checking out FastBin! However, in order to securely load and decrypt, you will have to enable JavaScript.


Brian Bier
|
HOME
ABOUT
PORTFOLIO
BLOG
CONTACT
RESUME
Full Stack Developer
UX Designer
Programming Guru
Projects
Upodcast.me
Brian Bier
Take a look
Sasha Ostojic
Brian Bier
Take a look
Incase you missed it
Brian Bier
Take a look
IMDB Clone
Brian Bier
Take a look
Got a site that needs help?
CALL ME TODAY


[gecode-users] Slight modification of examples/money.cc that reveals a bug
Olof Sivertsson
osivertsson at gmail.com
Tue Feb 28 16:15:42 CET 2006
Hello,
I think I've stumbled across a bug that is quite easy to reproduce by doing
a slight modification of examples/money.cc.
Without modification running examples/money will print:
9 5 6 7 1 0 8 2
which is the values of the letters in the array [s,e,n,d,m,o,r,y].
Now let's force s = 9 by adding the following to Money(const Options& opt)
post(this, s == 9);
Recompile & rerun:
> make compileexamples
> examples/money
...
9 5 6 7 1 0 8 2
No change - it works like it should. But now let's do a small change, and
instead post the following constraint:
post(this, 9 == s);
If we recompile and rerun it now we get no solution, so somethings broken. I
bet it's a sign error since if we change it to:
post(this, -9 == s);
it works!
Hopefully this will lead you straight to the bug and a trivial fix.
Cheers,
Olof
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.ps.uni-sb.de/pipermail/users/attachments/20060228/59250226/attachment.htm>
More information about the gecode-users
mailing list


#include <stFastMapper.h>
Public Member Functions | |
| stFastMapImage (ObjectType *object, int dim, bool ownership=false) | |
| virtual | ~stFastMapImage () |
| const ObjectType * | GetObject () |
| void | SetObject (ObjectType *obj, bool ownership=false) |
| double & | operator[] (int idx) |
| void | GetCoords (double *coords) |
| const double * | GetCoords () |
| void | SetCoords (const double *coords) |
| void | SetCoordsTo (double value=0) |
| int | GetDimension () |
| stFastMapImage * | Clone () |
This class template is a utility class for stFastMapper which can hold an object image (the projected coordinates and a reference to the original object). It is used during the pivot election and to hold the pivot information inside an instance of stFastMapper.
The object type (ObjectType) must implement at least a method named Clone() which can create a perfect clone of the object.
This class template is a companion of stFastMapper and is not supposed to be used by other classes.
Definition at line 127 of file stFastMapper.h.
|
inline |
Creates a new instance of this class.
| object | The associated object. |
| dim | The number of dimensions. |
| ownership | A flag which tells this image to take ownership of the given object. |
Definition at line 137 of file stFastMapper.h.
Referenced by stFastMapImage< ObjectType >::Clone().
|
inlinevirtual |
Disposes this instance. The object associated with this image will be deleted if this instance owns it.
Definition at line 150 of file stFastMapper.h.
|
inline |
This method creates a perfect clone of this object.
Definition at line 236 of file stFastMapper.h.
References stFastMapImage< ObjectType >::SetCoords(), and stFastMapImage< ObjectType >::stFastMapImage().
|
inline |
Returns all coordinates. The output vector must have at least dim positions.
| coords | The output vector. |
Definition at line 198 of file stFastMapper.h.
|
inline |
Returns all coordinates in a vector. Do NOT modify these values using this method.
Definition at line 206 of file stFastMapper.h.
|
inline |
Returns the number of dimensions of this image.
Definition at line 229 of file stFastMapper.h.
|
inline |
Returns the object associated with this instance.
Definition at line 164 of file stFastMapper.h.
|
inline |
This operator allows the direct access to each image coordinate.
Definition at line 188 of file stFastMapper.h.
|
inline |
Sets all coordinates of this image using the input vector of values.
| coords | The input vector. |
Definition at line 215 of file stFastMapper.h.
Referenced by stFastMapImage< ObjectType >::Clone().
| void stFastMapImage< ObjectType >::SetCoordsTo | ( | double | value = 0 | ) |
Sets all coordinates to a given value.
| value | A value which will be used to initialize all coordinates. |
|
inline |
Changes the object. Used for recycle purposes.
| obj | The new object. |
| ownership | A flag which tells this image to take ownership of the given object. |
Definition at line 175 of file stFastMapper.h.