UKCloud Limited (“UKC”) and Virtual Infrastructure Group Limited (“VIG”) (together “the Companies”) – in Compulsory Liquidation
On 25 October 2022, the Companies were placed into Liquidation with the Official Receiver appointed as Liquidator and J Robinson and A M Hudson simultaneously appointed as Special Managers to manage the liquidation process on behalf of the Official Receiver.
Further information regarding the Liquidations can be found here: https://www.gov.uk/government/news/virtual-infrastructure-group-limited-and-ukcloud-limited-information-for-creditors-and-interested-parties
Contact details:
For any general queries relating to the Liquidations please email<EMAIL_ADDRESS>For customer related queries please email<EMAIL_ADDRESS>For supplier related queries please email<EMAIL_ADDRESS>Patching as a Service FAQs
Service
What is the service?
As customer solutions start to expand beyond their own data centres into embracing the benefits of leveraging multi-cloud, the day-to-day management and control of operating system (OS) patching across various technology stacks and locations become highly complex and resource intensive. Patching as a Service helps to reduce these challenges, whilst increasing your level of trust and confidence in the security health of any operating systems deployed as part of your solution stack.
With Patching as a Service, we provide routine OS patch management for compute instances deployed across UKCloud's multi-cloud. It helps you accelerate business value by removing the burden of basic OS hygiene, letting you focus your attention above the operating system.
What technologies do you use?
Built on the capabilities of our Managed Monitoring as a Service, UKCloud uses a combination of patches released directly by the vendor coupled with Ivanti Security Controls, a market-leading enterprise patch management and control solution, as the key technologies to deliver Patching as a Service.
Do I get access to a dashboard or reports?
As it is assumed UKCloud will manage all patching aspects as part of this service, there are currently no dashboard or reporting features for customers to access.
How do I get alerted on events?
All alerting will be performed via a support ticket or any special escalation process agreed between UKCloud and the customer.
How does UKCloud provide urgent maintenance notifications and incident reports?
We do not envisage any scenarios that prevent UKCloud in meeting the availability KPI defined within the Service Scope for this service.
Current Service Status reports and past incident reports are published on the UKCloud Service Status page.
What type of patches will be applied?
For Microsoft operating systems we'll scan for and recommend applying any patches classified by Microsoft as being 'Security Patches' of any severity (Critical to Low). Additional patches for items such as 'Feature Packs' or general driver updates are excluded from this service.
For Linux operating systems we'll scan for and recommend applying any packages that have an update available in the repositories specified within the guest operating system (equivalent to manually running a yum update list
command in RHEL or CentOS).
How does the patching cycle work?
OS vendors such as Microsoft release updates for their operating systems and associated packages in two ways:
To a specific schedule: "Patch Tuesday" / "Update Tuesday"
Ad-hoc, throughout the course of each month
As such, any patching cycle needs to take into consideration the two potential release mechanisms for updates. For our Patching as a Service, we use a batch approach with specific cut-off dates, which we consider to be the best approach to maintain consistency in UKCloud's service, provide a good level of responsiveness to patch releases and also to insulate sufficiently against erroneous upstream vendor patches or errors. This means that there's a specific cut-off after which point any patches released will not be included in that cycle (see Windows Patches vs Linux Package Updates). Rather than being a limitation, this ensures that we can correctly and efficiently track activities when dealing with customer VMs.
UKCloud's patching schedule works as detailed below.
2nd Tuesday of the month (Patch Tuesday)
- Microsoft releases critical patches for the month.
3rd Tuesday of the month
- Microsoft releases further patches addressing any issues with patches on Patch Tuesday.
3rd Wednesday of the month
Between 00:00 and 06:00 the patching service scans all endpoints for vulnerabilities and updates the patching information in the database.
At 12:00 the patching service ceates a Patching Notification Service Request for each patch group for the customer's visibility. A notification email is also sent.
Customers have from the time they receive the notification to the beginning of their patch window to notify UKCloud Support if they want to opt out of any patches.
3rd Thursday of the month
Patching commences for all patch groups on the below schedule. Steps taken during patching run are:
Devices are put into maintenance in the UKCloud monitoring system. Managed Monitoring tickets will not be raised against devices in their patching window to avoid false positives.
UKCloud's Linux patching system connects to all customer devices via ssh and commences applying all patches that have not been identified as opted out or permanently excluded.
- A report is attached to the previously raised Service Request and the Service Request is updated with the Linux patching outcome.
UKCloud's Windows patching system connects performs a scan against each customer device for the patches that have not been identified as opted out or permanently excluded.
It performs a patch deployment against each scan that found patches to apply.
A report is attached to the previously raised Service Request and the Service Request is updated with the Windows patching outcome.
If both the Windows and Linux patching outcomed are set to Completed successfully the Service Request is closed.
If either operating system outcome is set to Completed with errors or Failed the customer is updated with an email. The customer must investigate errors and contact UKCloud Support if they require assistance.
If there has been no update after two weeks, the Service Request times out and UKCloud Support is alerted that not all patching was completed for the customer's patch group. UKCloud Support will try to contact the customer before ultimately closing the Patching Notification Service Request.
5th Wednesday of the month
Between 00:00 and 06:00 the patching service scans all endpoints for vulnerabilities and updates the patching information in the database.
At 12:00 the patching service creates Service Requests for each patch group in My Calls. A notification email is sent.
Customers have from the time they receive the notification to the beginning of their patch window to notify UKCloud Support if they want to opt out of any patches.
5th Thursday of the month
- Patching commences again for all patch groups.
You can see the full schedule in the table below.
| Group | Day | Time | Cycle | Group | Day | Time | Cycle | |
|---|---|---|---|---|---|---|---|---|
| A | Thurs | 00:00 - 05:00 | 1 | AA | Thurs | 00:00 - 05:00 | 2 | |
| B | Thurs | 06:00 - 11:00 | 1 | AB | Thurs | 06:00 - 11:00 | 2 | |
| C | Thurs | 12:00 - 17:00 | 1 | AC | Thurs | 12:00 - 17:00 | 2 | |
| D | Thurs | 18:00 - 23:00 | 1 | AD | Thurs | 18:00 - 23:00 | 2 | |
| E | Fri | 00:00 - 05:00 | 1 | AE | Fri | 00:00 - 05:00 | 2 | |
| F | Fri | 06:00 - 11:00 | 1 | AF | Fri | 06:00 - 11:00 | 2 | |
| G | Fri | 12:00 - 17:00 | 1 | AG | Fri | 12:00 - 17:00 | 2 | |
| H | Fri | 18:00 - 23:00 | 1 | AH | Fri | 18:00 - 23:00 | 2 | |
| I | Sat | 00:00 - 05:00 | 1 | AI | Sat | 00:00 - 05:00 | 2 | |
| J | Sat | 06:00 - 11:00 | 1 | AJ | Sat | 06:00 - 11:00 | 2 | |
| K | Sat | 12:00 - 17:00 | 1 | AK | Sat | 12:00 - 17:00 | 2 | |
| L | Sat | 18:00 - 23:00 | 1 | AL | Sat | 18:00 - 23:00 | 2 | |
| M | Sun | 00:00 - 05:00 | 1 | AM | Sun | 00:00 - 05:00 | 2 | |
| N | Sun | 06:00 - 11:00 | 1 | AN | Sun | 06:00 - 11:00 | 2 | |
| O | Sun | 12:00 - 17:00 | 1 | AO | Sun | 12:00 - 17:00 | 2 | |
| P | Sun | 18:00 - 23:00 | 1 | AP | Sun | 18:00 - 23:00 | 2 | |
| Q | Mon | 00:00 - 05:00 | 1 | AQ | Mon | 00:00 - 05:00 | 2 | |
| R | Mon | 06:00 - 11:00 | 1 | AR | Mon | 06:00 - 11:00 | 2 | |
| S | Mon | 12:00 - 17:00 | 1 | AS | Mon | 12:00 - 17:00 | 2 | |
| T | Mon | 18:00 - 23:00 | 1 | AT | Mon | 18:00 - 23:00 | 2 | |
| U | Tues | 00:00 - 05:00 | 1 | AU | Tues | 00:00 - 05:00 | 2 | |
| V | Tues | 06:00 - 11:00 | 1 | AV | Tues | 06:00 - 11:00 | 2 | |
| W | Tues | 12:00 - 17:00 | 1 | AW | Tues | 12:00 - 17:00 | 2 | |
| X | Tues | 18:00 - 23:00 | 1 | AX | Tues | 18:00 - 23:00 | 2 |
Note
The patching cycle is two weeks long (the dates in the table above show the day number from the beginning of each cycle rather than the day of the month).
the 5th and 6th weeks of the month may occur at the beginning of the following week.
Customers are advised to use Device Groups A to X to test patches on VMs within a representative non-production environment.
Customers are advised to use Device Groups AA to AX to apply patches on VMs within their production environments on the basis that no issues have been identified.
Does my server/virtual machine need to be powered-on for the patching service to run?
For the patching service to operate correctly, VMs must be in a powered-on and available state throughout the patching schedule selected by the customer. The patching service is currently unable to change the power state of any server or virtual machine ahead of the selected patching schedule. Any server or VM in a powered-off state during the selected patching schedule will not be scanned for missing patches or have missing patches applied.
Why are virtual appliances excluded from the scope of this service?
Due to the tightly coupled nature of the operating system and application created when a vendor releases a virtual appliance, there's a high risk of corruption of the virtual appliance and its service availability should standard operating system patches be applied. This risk becomes even greater should a vendor modify the operating system a virtual appliance is based upon, something which is outside of UKCloud's control.
How do I recover from applying a patch that may have had an adverse effect on my environment?
Before applying patches within a production environment, it is the customer's responsibility to test those patches in a representative non-production environment to detect any potential adverse impact within the production environment.
For customers using Patching as a Service on our UKCloud for VMware platform, UKCloud will endeavour to create a recovery point of a VM immediately prior to applying patches. As we cannot guarantee the creation of a recovery point (for example, open files may block the creation of a recovery point), UKCloud provides the following options for customers to choose from to determine how Patching as a Service should proceed should the creation of a recovery point fail:
Hard (default option): The patching system will attempt to snapshot the virtual machine prior to applying any patches. If the snapshot is unsuccessful, patching will not proceed.
Soft: The patching system will attempt to snapshot the virtual machine prior to applying any patches. If the snapshot is unsuccessful, patching will still proceed.
Disabled: The patching system will not attempt to snapshot the virtual machine prior to applying any patches. This option is for virtual machines not running on the UKCloud VMware platform, or for virtual machines that are unable to have snapshots taken.
UKCloud will retain any successful recovery points for a maximum of 48 hours, therefore it is essential that customers check for any potential issues within 48 hours of any patches being applied.
Windows patches vs Linux package updates
There are major differences between the way that Microsoft releases patches for its Windows OSs and the way that most Linux distribution vendors provide updates. The key difference is:
Microsoft releases patches for their OS (which can be either critical or recommended)
Linux vendors release patches for individual packages that make up the OS.
To make best use of existing tooling within those Linux distributions, our service will update a package rather than an individual patch (for example, the equivalent of running yum update httpd
, which will include all patches related to the httpd package).
What if the application of a patch fails?
Should the application of a patch fail, the UKCloud patching system will update the Patching Notification Service Request with the outcome. The post-patch report attached to the request will include information about which devices have failed to patch, and why. After two weeks of no updates on a Service Request related to a patching group that has failed to patch, UKCloud Support will make best efforts to contact the customer once more before closing the Service Request.
It is the customer's responsibility to ensure their devices are successfully patching and to investigate any failures. UKCloud Support is available to provide assistance with troubleshooting and investigation if required, and can be contacted by updating the Patching Notification Service Request.
What if I require additional patches outside of my standard patching cycle?
The customer is responsible for notifying UKCloud of any patches (including critical patches) they require to be applied outside of their standard patching cycle via the My Calls section of the Portal. UKCloud will not proactively scan for or apply any patches outside of a customer's standard patching cycle. Should a customer raise a Service Request for patches to be applied outside of their standard patching cycle, our standard Service Request resolution target will apply, with the clock starting from the point we receive the Service Request and a vendor released patch being made publicly available.
Prerequisites
Operating systems
UKCloud will only support in-life, vendor-backed operating systems, such as Microsoft Windows, Red Hat Enterprise Linux and Canonical Ubuntu. Customer-built operating systems (for example Gentoo Linux custom build) are unsupported by this service.
Customer environment configuration
General - For UKCloud to correctly scan for and apply OS patches, the customer must agree to enabling and configuring the Server Message Block (SMB) protocol by ensuring ports 445 and 139 (both over TCP) are open on all virtual machines and associated operating systems opted-in to this service.
Linux Only - To provide secure communication between UKCloud's centralised patch management platform and a customer's virtual machine, we use key-based SSH sessions, reducing the need to share credentials. As such, we'll need the following prerequisites to be fulfilled on each virtual machine subscribing to this service:
An allow rule for port 22 (SSH) on your virtual machine's firewall (for example, iptables).
A local user account with passwordless sudo access. You'll need to share details of this account with UKCloud when onboarding to this service.
UKCloud will provide you with a public SSH key during the onboarding process. You'll need to append this key to the
authorized_keys
file, located at~/.ssh/authorized_keys
within the home directory of the user you created in the step above.
Management
How do I access my patching environment?
As it is assumed UKCloud will manage all OS patching aspects as part of this service, there is currently no environment for customers to access.
How do I manage my services?
As it is assumed UKCloud will manage all OS patching aspects as part of this service, there is currently no way for customers to directly manage this service.
Any changes to patch scheduling after customers have selected their patch windows will require consultation and should be requested initially by raising a ticket within the My Calls section of the Portal. Alternatively, you can contact UKCloud Support by phone or email.
Support
How do I raise a support ticket?
The secure online UKCloud Portal provides most common service management functionality. You can raise tickets within the My Calls section of the Portal. Alternatively, you can contact UKCloud Support by phone or email.
What are your service maintenance windows?
See Understanding UKCloud service maintenance windows.
Can UKCloud provide notifications for service status and maintenance?
Yes, you can subscribe to the UKCloud Service Status page to be alerted to service status and maintenance notifications. Notifications can be via email, text, webhook or Atom/RSS feed. For more information, see How to subscribe to service status notifications.
Onboarding
How can I get started with the service?
Within 5 business days of accepting an order, UKCloud will create the customer's patching environment and commence patching any environments in line with the Device Group schedules they have opted-in to.
Is there a free trial?
Due to the initial complexity of configuring the patching environment, UKCloud is currently unable to offer a free trial of this service.
Billing and legal
What is the smallest unit of time I will be billed for?
The minimum unit of time for use is monthly. Part months will be rounded up.
Will I be charged for resources in a 'shut off' state?
UKCloud will continue to charge for any resources that you have elected to be monitored regardless of the state they're in. To stop any charges, you'll need to raise a ticket within the My Calls section of the Portal. Alternatively, you can contact UKCloud Support by phone or email.
How can I view billing information?
Online visibility of real-time or historic billing information is currently not available as a service provided by UKCloud.
How can I pay for the services?
Payment for UKCloud services can be made by direct bank transfer (BACS/CHAPS). You can find bank details for payment on your invoice.
What are the termination fees?
There are no termination costs for this service.
Security
How is any data related to this service kept secure?
UKCloud will deploy and manage all agents used to deliver this service, with all traffic being passed via HTTPS.
Any customer data that UKCloud collects as part of this service will never leave a UKCloud data centre or be shared with a third party.
Is there a protective monitoring service?
Protective monitoring is included with all our IaaS platforms at the hypervisor level and below and follows GPG 13. For more information, see Protective Monitoring from UKCloud.
If you require protective monitoring services above the hypervisor we offer our Security Operations Service to monitor your virtual estate (additional charges apply). For more information, see the Security Operations Service Service Definition.
Feedback
If you find a problem with this article, click Improve this Doc to make the change yourself or raise an issue in GitHub. If you have an idea for how we could improve any of our services, send an email to feedback@ukcloud.com.


 JS can really test you as a stable minded human being!  But this week has been good and the more time I spend looking at the JS code the more comfortable I feel. I should just look at it 24/7, right?
This week I had a couple blocks, one being a pretty simple problem in that I had a syntax error when trying to access a property that was first in an array. I used a couple techniques to try and solve this problem by googling, then eventually reaching out to peers in my cohort as I could not finding the right way to call it. I felt pretty good during this process as I knew all I needed to do was find out how to call it as there were two properties in that array and we only wanted to modify the first one in the DOM. Andrew ended up finding the right syntax by googling, he passed the link on to me (thank you!) in which I corrected my syntax and broke out of the block.
From that block experience I learned that if it is a syntax error I should google a bit longer and read deeper as the answer is usually there right in front of you, sometimes you just need to have a proper look. I should do the same for my keys and sunglasses too!
Even though I only used two for that block there are many more problem solving techniques that are good to use when you are blocked or just generally trying to solve a problem.
Pseudocode: I'm feeling semi-confident using Pseudocode. When doing the assingments this week the step-by-step process that we were supposed to do to output a code was written pretty well and I feel I understood what was being asked of us, but found in tough in later assignments to put it into code. I think it was mostly just not knowing correct syntax so hoping that'll come right later with more us of JS.
Trying something: I'm pretty good at trying to write the code. I usually have a pretty good feel of what is needed, it's just getting it correct which is proving a different beast.
Rubber ducky method: I haven't yet tried this method but I usually talk to myself, so talking to an inanimate object isn't far from my realm of craziness.
Reading error messages: To be honest I'm not too good at reading error messages in the terminal. It's quite overwhelming seeing all that code come up but is definantly something that I should not shy away from. In Codeacdemy I'm not bad with their error messages as it gives you a few different options as to what the error my be.
Console.logging: I haven't really used console.log this week in our assignments but again is something I should use. Especially when trying to call array properties.
Googling: I do Google quite a bit when I'm blocked but I find myself just skim reading at times. If I read properly I may find the answer I am looking for faster. With that being said I have become unblocked quite a few times just by using Google so that surely means I'm not terrible at it!
Asking your peers for help: Now this I am good at! Maybe too good.. Sometimes it feels like I'm the only one who gets blocked by looking at our slack history, but that's OK. Like I said earlier, if I used Google and the other techniques better I may not need to ask on our Slack group so much. Sometimes it feels like the easy way out.
Asking your coaches for help: See above!
Improving your process with reflection: I am pretty decent at reflecting back on the block I just had and what I could have done better or what I did well. As I said above I do need to apply myself better when intially becoming blocked, which means trying harder to find the issues myself or looking through my notes as the answer is most likely in there. I feel I am becoming better with my mindset when I am blocked as I know there is always an answer to the issue and it isn't far away!
Array.prototype.map( ): takes a selected array and makes a new array with a set of things you want to perform on the original. The array you used the map on stays the same and a new one is made.
Array.prototype.filter( ): basically doesn't let things into an array that do not met set requirements to be inside the array. e.g if only strings are allowed inside the array there will no numbers inside when the new array is logged out.
Array.prototype.reduce( ): takes a set of things that is applied to it and returns a single value instead of the whole array which may have 2 or more values.


02/14/13
Rancho Gordo New World Specialty Food, a Napa-based specialty food company has announced plans to open a retail space at the San Francisco Ferry Plaza Building Marketplace, home of Bay Area food legends like Acme Bread, Cowgirl Creamery and the Slanted Door. The space will feature Rancho Gordo’s collection of over 25 heirloom and heritage dry beans as well as other indigenous “New World” products such as chilies and corn. The Rancho Gordo store is planned to open summer 2013.
Founder Steve Sando says, “We’ve been selling at the Saturday farmers market for about eight years. The move inside makes sense. Without the farmers market restrictions of California-grown, we can show off a lot of the other projects we’re working on, especially those with farmers in Mexico and Honduras.” In addition to the beans, Rancho Gordo plans to sell clay cooking pots, chilies, tortillas made from heirloom corn, stoneground cacao and many other items under the Rancho Gordo brand.
Rancho Gordo started as a hobby in Sando᾿s Napa backyard until he started selling his heirloom beans at Bay Area farmers markets. The colorful beans caught the eye of chefs like Thomas Keller (French Laundry), Annie Somerville (Greens) and Stuart Brioza (State Bird Provisions) and home cooks alike. Rancho Gordo grows 25 varieties in California and imports about 10 varieties a year from small agriculture cooperatives and farmers in Mexico through their Rancho Gordo-Xoxoc Project. Sando has written two books on heirloom beans, including Heirloom Beans: Recipes from Rancho Gordo from Chronicle Books, now in its third printing with sales of 24,000 copies.
The new store will be the company’s second retail outlet. The Napa headquarters for Rancho Gordo houses offices, a warehouse and a small retail store. Rancho Gordo products are also available on the website, ranchogordo.com, and at wholesale to better stores and restaurants throughout the country.


This website uses cookies to ensure you get the best experience. Learn more about DOAJ’s privacy policy.
Hide this message


| Class | Description |
|---|---|
| BarrierStream | |
| DeferredStream<T> |
A
Publisher supplier that will call the passed supplier on each subscribe call. |
| ErrorStream<O,T extends Throwable> |
A Stream that emits a sigle error signal.
|
| FutureStream<T> |
A Stream that emits a result of a
Future and then complete. |
| GroupedStream<K,T> |
The Stream will complete or fail whever the parent groupBy action terminates itself.
|
| IterableStream<T> |
A Stream that emits
Iterable values one by one and then complete. |
| LiftStream<O,V> |
A Stream wrapper that defers a parent stream subscription to the child action subscribe() call.
|
| MapStream<K,V> |
A SubscribableMap is an event-driven Map that signals logged operations to its subscribers.
|
| MapStream.MutableSignal<K,V> | |
| MapStream.Signal<K,V> | |
| PeriodicTimerStream |
A Stream that emits
0 after an initial delay and ever incrementing long counter if the period argument is
specified. |
| PublisherStream<T> |
A
Publisher wrapper that takes care of lazy subscribing. |
| RangeStream |
A Stream that emits N
Long from the inclusive start value defined to the inclusive end and then
complete. |
| SingleTimerStream |
A Stream that emits
0 after an initial delay and then complete |
| SingleValueStream<T> |
A Stream that emits only one value and then complete.
|
| SupplierStream<T> |
A Stream that returns the result from
Supplier.get() everytime it is requested via
Subscription.request(long) . |
| Enum | Description |
|---|---|
| MapStream.Operation |


I'm {{ .Site.Params.author }}, a peppy computational biologist. I'm currently a Statistical Geneticist at Bridge Bio, where I use functional genomics to prioritize targets for rare genetic diseases.
My passions include open source software, open science, molecular sequence data, tea, and Beyoncé.
My goals in life are to understand how a cell works and empower others through programming.
{{ partial "social.html" . }}
While you're here, if you're a biologist that wants to learn programming, or a programmer that wants to learn biology, here is the active learning textbook for you.


Improvements in task scheduling for batch workloads in Apache Flink 1.12
December 2, 2020 - Andrey ZagrebinThe Flink community has been working for some time on making Flink a truly unified batch and stream processing system. Achieving this involves touching a lot of different components of the Flink stack, from the user-facing APIs all the way to low-level operator processes such as task scheduling. In this blogpost, we’ll take a closer look at how far the community has come in improving scheduling for batch workloads, why this matters and what you can expect in the Flink 1.12 release with the new pipelined region scheduler.
Towards unified scheduling #
Flink has an internal scheduler to distribute work to all available cluster nodes, taking resource utilization, state locality and recovery into account. How do you write a scheduler for a unified batch and streaming system? To answer this question, let’s first have a look into the high-level differences between batch and streaming scheduling requirements.
Streaming #
Streaming jobs usually require that all operator subtasks are running in parallel at the same time, for an indefinite time. Therefore, all the required resources to run these jobs have to be provided upfront, and all operator subtasks must be deployed at once.
Flink: Streaming job example
Because there are no finite intermediate results, a streaming job always has to be restarted fully from a checkpoint or a savepoint in case of failure.
Batch #
In contrast to streaming jobs, batch jobs usually consist of one or more stages that can have dependencies between them. Each stage will only run for a finite amount of time and produce some finite output (i.e. at some point, the batch job will be finished). Independent stages can run in parallel to improve execution time, but for cases where there are dependencies between stages, a stage may have to wait for upstream results to be produced before it can run. These are called blocking results, and in this case stages cannot run in parallel.
Flink: Batch job example
As an example, in the figure above Stage 0 and Stage 1 can run simultaneously, as there is no dependency between them. Stage 3, on the other hand, can only be scheduled once both its inputs are available. There are a few implications from this:
-
(a) You can use available resources more efficiently by only scheduling stages that have data to perform work;
-
(b) You can use this mechanism also for failover: if a stage fails, it can be restarted individually, without recomputing the results of other stages.
Scheduling Strategies in Flink before 1.12 #
Given these differences, a unified scheduler would have to be good at resource management for each individual stage, be it finite (batch) or infinite (streaming), and also across multiple stages. The existing scheduling strategies in older Flink versions up to 1.11 have been largely designed to address these concerns separately.
“All at once (Eager)”
This strategy is the simplest: Flink just tries to allocate resources and deploy all subtasks at once. Up to Flink 1.11, this is the scheduling strategy used for all streaming jobs. For batch jobs, using “all at once” scheduling would lead to suboptimal resource utilization, since it’s unlikely that such jobs would require all resources upfront, and any resources allocated to subtasks that could not run at a given moment would be idle and therefore wasted.
“Lazy from sources”
To account for blocking results and make sure that no consumer is deployed before their respective producers are finished, Flink provides a different scheduling strategy for batch workloads. “Lazy from sources” scheduling deploys subtasks only once all their inputs are ready. This strategy operates on each subtask individually; it does not identify all subtasks which can (or have to) run at the same time.
A practical example #
Let’s take a closer look at the specific case of batch jobs, using as motivation a simple SQL query:
CREATE TABLE customers (
customerId int,
name varchar(255)
);
CREATE TABLE orders (
orderId int,
orderCustomerId int
);
--fill tables with data
SELECT customerId, name
FROM customers, orders
WHERE customerId = orderCustomerId
Assume that two tables were created in some database: the customers
table is relatively small and fits into the local memory (or also on disk). The orders
table is bigger, as it contains all orders created by customers, and doesn’t fit in memory. To enrich the orders with the customer name, you have to join these two tables. There are basically two stages in this batch job:
- Load the complete
customers
table into a local map:(customerId, name)
; because this table is smaller, - Process the
orders
table record by record, enriching it with thename
value from the map.
Executing the job #
The batch job described above will have three operators. For simplicity, each operator is represented with a parallelism of 1, so the resulting ExecutionGraph will consist of three subtasks: A, B and C.
- A: load full
customers
table - B: load
orders
table record by record in a streaming (pipelined) fashion - C: join order table records with the loaded customer table
This translates into A and C being connected with a blocking data exchange,
because the customers
table needs to be loaded locally (A) before we start processing the order table (B).
B and C are connected with a pipelined data exchange,
because the consumer (C) can run as soon as the first result records from B have been produced.
You can think of B->C as a finite streaming job. It’s then possible to identify two separate stages within the ExecutionGraph: A and B->C.
Flink: SQL Join job example
Scheduling Limitations #
Imagine that the cluster this job will run in has only one slot and can therefore only execute one subtask. If Flink deploys B chained with C first into this one slot (as B and C are connected with a pipelined edge), C cannot run because A has not produced its blocking result yet. Flink will try to deploy A and the job will fail, because there are no more slots. If there were two slots available, Flink would be able to deploy A and the job would eventually succeed. Nonetheless, the resources of the first slot occupied by B and C would be wasted while A was running.
Both scheduling strategies available as of Flink 1.11 (“all at once” and “lazy from source”) would be affected by these limitations. What would be the optimal approach? In this case, if A was deployed first, then B and C could also complete afterwards using the same slot. The job would succeed even if only a single slot was available.
Last but not least, let’s consider what happens in the case of failover: if the processing of the orders
table fails (B->C),
then we do not have to reload the customer table (A); we only need to restart B->C. This did not work prior to Flink 1.9.
To satisfy the scheduling requirements for batch and streaming and overcome these limitations, the Flink community has worked on a new unified scheduling and failover strategy that is suitable for both types of workloads: pipelined region scheduling.
The new pipelined region scheduling #
As you read in the previous introductory sections, an optimal scheduler should efficiently allocate resources for the sub-stages of the pipeline, finite or infinite, running in a streaming fashion. Those stages are called pipelined regions in Flink. In this section, we will take a deeper dive into pipelined region scheduling and failover.
Pipelined regions #
The new scheduling strategy analyses the ExecutionGraph before starting the subtask deployment in order to identify its pipelined regions. A pipelined region is a subset of subtasks in the ExecutionGraph connected by pipelined data exchanges. Subtasks from different pipelined regions are connected only by blocking data exchanges. The depicted example of an ExecutionGraph has four pipelined regions and subtasks, A to H:
Flink: Pipelined regions
Why do we need the pipelined region? Within the pipelined region all consumers have to constantly consume the produced results to not block the producers and avoid backpressure. Hence, all subtasks of a pipelined region have to be scheduled, restarted in case of failure and run at the same time.
Pipelined region scheduling strategy #
Once the pipelined regions are identified, each region is scheduled only when all the regions it depends on (i.e. its inputs), have produced their blocking results (for the depicted graph: R2 and R3 after R1; R4 after R2 and R3). If the JobManager has enough resources available, it will try to run as many schedulable pipelined regions in parallel as possible. The subtasks of a pipelined region are either successfully deployed all at once or none at all. The job fails if there are not enough resources to run any of its pipelined regions. You can read more about this effort in the original FLIP-119 proposal.
Failover strategy #
As mentioned before, only certain regions are running at the same time. Others have already produced their blocking results. The results are stored locally in TaskManagers where the corresponding subtasks run. If a currently running region fails, it gets restarted to consume its inputs again. If some input results got lost (e.g. the hosting TaskManager failed as well), Flink will rerun their producing regions. You can read more about this effort in the user documentation and the original FLIP-1 proposal.
Benefits #
Run any batch job, possibly with limited resources
The subtasks of a pipelined region are deployed only when all necessary conditions for their success are fulfilled: inputs are ready and all needed resources are allocated. Hence, the batch job never gets stuck without notifying the user. The job either eventually finishes or fails after a timeout.
Depending on how the subtasks are allowed to share slots, it is often the case that the whole pipelined region can run within one slot, making it generally possible to run the whole batch job with only a single slot. At the same time, if the cluster provides more resources, Flink will run as many regions as possible in parallel to improve the overall job performance.
No resource waste
As mentioned in the definition of pipelined region, all its subtasks have to run simultaneously. The subtasks of other regions either cannot or do not have to run at the same time. This means that a pipelined region is the minimum subgraph of a batch job’s ExecutionGraph that has to be scheduled at once. There is no way to run the job with fewer resources than needed to run the largest region, and so there can be no resource waste.
Conclusion #
Scheduling is a fundamental component of the Flink stack. In this blogpost, we recapped how scheduling affects resource utilization and failover as a part of the user experience. We described the limitations of Flink’s old scheduler and introduced a new approach to tackle them: the pipelined region scheduler, which ships with Flink 1.12. The blogpost also explained how pipelined region failover (introduced in Flink 1.11) works.
Stay tuned for more improvements to scheduling in upcoming releases. If you have any suggestions or questions for the community, we encourage you to sign up to the Apache Flink mailing lists and become part of the discussion.
Appendix #
What is scheduling? #
ExecutionGraph #
A Flink job is a pipeline of connected operators to process data. Together, the operators form a JobGraph. Each operator has a certain number of subtasks executed in parallel. The subtask is the actual execution unit in Flink. Each subtask can consume user records from other subtasks (inputs), process them and produce records for further consumption by other subtasks (outputs) down the stream. There are source subtasks without inputs and sink subtasks without outputs. Hence, the subtasks form the nodes of the ExecutionGraph.
Intermediate results #
There are also two major data-exchange types to produce and consume results by operators and their subtasks: pipelined and blocking. They are basically types of edges in the ExecutionGraph.
A pipelined result can be consumed record by record. This means that the consumer can already run once the first result records have been produced. A pipelined result can be a never ending output of records, e.g. in case of a streaming job.
A blocking result can be consumed only when its production is done. Hence, the blocking result is always finite and the consumer of the blocking result can run only when the producer has finished its execution.
Slots and resources #
A TaskManager instance has a certain number of virtual slots. Each slot represents a certain part of the TaskManager’s physical resources to run the operator subtasks, and each subtask is deployed into a slot of the TaskManager. A slot can run multiple subtasks from different operators at the same time, usually chained together.
Scheduling strategy #
Scheduling in Flink is a process of searching for and allocating appropriate resources (slots) from the TaskManagers to run the subtasks and produce results. The scheduling strategy reacts on scheduling events (like start job, subtask failed or finished etc) to decide which subtask to deploy next.
For instance, it does not make sense to schedule subtasks whose inputs are not ready to consume yet to avoid wasting resources. Another example is to schedule subtasks which are connected with pipelined edges together, to avoid deadlocks caused by backpressure.


Hey, we're still preparing to launch. Subcribe us to get notified.
By clicking 'Notify Me' you agree to our Privacy Policy


Welcome to birdThing!
This app lets you to practice your bird recognising skills.
Choose your type of clue and guess which bird it refers to!
Lets get
Started!


--- layout: default ---
As a student organization, we have a strong presence on campus and a strong backing from the international IEEE organization. On campus, we have technical speakers present at our meetings, company representatives talk about their technology and recruit our students, social events, and community service events. Also, we work with HKN to manage and staff the dB cafe, a food stand that provides food at a low cost to students, faculty, and anyone else who comes by our stand.


{% extends "layout.html" %} {% block content %}
Submit log data in GPX format and Trail Route Analytics let you analyze your climbing/trail running/speed hiking activity.
Tool only accepts GPX formatted file at a momemt. You can export GPX file out from your GPS device/application(e.g. 'Export' menu of MyTracks), or download from various websites where route data is available.
Please note that tool does not support GPX file containing multiple days/set of activities, so you should split your activities accordingly.
{% endblock %}


-ac_notarypay=<sats per block paid to notaries>
The code tracks how long it has been from 1 notarization to the next pays the above amount multiplied by the number of blocks, divided by the number of notaries who notarized the block. This allows notaries pay to be part of the emission schedule without making it vary. And also stops notaries notarizing faster to increase supply.
Basically what ever your -ac_reward is you would add the above value to that to get the total coin emission...


Student ID
Name
Computer Science
3741255
Jones, Martha
4077830
Pierce, Benjamin
5151701
Kirk, James
Russian Literature
3971244
Nim, Victor
Astrophysics
4100332
Petrov, Alexandra
8892377
Toyota, Hiroko


This is the most recent implementation report for the Verifiable Credentials Data Model specification.
Comments regarding this document are welcome. Please file issues directly on GitHub, or send them to<EMAIL_ADDRESS>(subscribe, archives).
The purpose of this document is to demonstrate that there are at least two interoperable implementations of processors that are capable of generating output that is conformant to the Verifiable Credentials Data Model.
The testing framework for the Verifiable Credentials Data Model executes the following process for every conformance statement in the Verifiable Credentials Data Model:
The results of the conformance testing are shown below:
%%%REPORTS%%%


asdasdasd
*** DIV 1: Some content here ***
*** DIV 2: Following right after div 1 ***
*** SPAN 1: Following right after div 2 ***
*** DIV 3: Following right after span 1
*** SPAN 2: INSIDE div 3 ***
Continue content of div 3 ***