tabyl
attributes to a data.frame.R/as_and_untabyl.R
A tabyl
is a data.frame containing counts of a variable or co-occurrences of two variables (a.k.a., a contingency table or crosstab). This specialized kind of data.frame has attributes that enable adorn_
functions to be called for precise formatting and presentation of results. E.g., display results as a mix of percentages, Ns, add totals rows or columns, rounding options, in the style of Microsoft Excel PivotTable.
A tabyl
can be the result of a call to janitor::tabyl()
, in which case these attributes are added automatically. This function adds tabyl
class attributes to a data.frame that isn't the result of a call to tabyl
but meets the requirements of a two-way tabyl:
1) First column contains values of variable 1
2) Column names 2:n are the values of variable 2
3) Numeric values in columns 2:n are counts of the co-occurrences of the two variables.*
* = this is the ideal form of a tabyl, but janitor's adorn_
functions tolerate and ignore non-numeric columns in positions 2:n.
For instance, the result of dplyr::count()
followed by tidyr::spread()
can be treated as a tabyl
.
The result of calling tabyl()
on a single variable is a special class of one-way tabyl; this function only pertains to the two-way tabyl.
as_tabyl(dat, axes = 2, row_var_name = NULL, col_var_name = NULL)
| dat | a data.frame with variable values in the first column and numeric values in all other columns. |
|---|---|
| axes | is this a two_way tabyl or a one_way tabyl? If this function is being called by a user, this should probably be "2". One-way tabyls are created by |
| row_var_name | (optional) the name of the variable in the row dimension; used by |
| col_var_name | (optional) the name of the variable in the column dimension; used by |
Returns the same data.frame, but with the additional class of "tabyl" and the attribute "core".
as_tabyl(mtcars)#> mpg cyl disp hp drat wt qsec vs am gear carb #> 21.0 6 160.0 110 3.90 2.620 16.46 0 1 4 4 #> 21.0 6 160.0 110 3.90 2.875 17.02 0 1 4 4 #> 22.8 4 108.0 93 3.85 2.320 18.61 1 1 4 1 #> 21.4 6 258.0 110 3.08 3.215 19.44 1 0 3 1 #> 18.7 8 360.0 175 3.15 3.440 17.02 0 0 3 2 #> 18.1 6 225.0 105 2.76 3.460 20.22 1 0 3 1 #> 14.3 8 360.0 245 3.21 3.570 15.84 0 0 3 4 #> 24.4 4 146.7 62 3.69 3.190 20.00 1 0 4 2 #> 22.8 4 140.8 95 3.92 3.150 22.90 1 0 4 2 #> 19.2 6 167.6 123 3.92 3.440 18.30 1 0 4 4 #> 17.8 6 167.6 123 3.92 3.440 18.90 1 0 4 4 #> 16.4 8 275.8 180 3.07 4.070 17.40 0 0 3 3 #> 17.3 8 275.8 180 3.07 3.730 17.60 0 0 3 3 #> 15.2 8 275.8 180 3.07 3.780 18.00 0 0 3 3 #> 10.4 8 472.0 205 2.93 5.250 17.98 0 0 3 4 #> 10.4 8 460.0 215 3.00 5.424 17.82 0 0 3 4 #> 14.7 8 440.0 230 3.23 5.345 17.42 0 0 3 4 #> 32.4 4 78.7 66 4.08 2.200 19.47 1 1 4 1 #> 30.4 4 75.7 52 4.93 1.615 18.52 1 1 4 2 #> 33.9 4 71.1 65 4.22 1.835 19.90 1 1 4 1 #> 21.5 4 120.1 97 3.70 2.465 20.01 1 0 3 1 #> 15.5 8 318.0 150 2.76 3.520 16.87 0 0 3 2 #> 15.2 8 304.0 150 3.15 3.435 17.30 0 0 3 2 #> 13.3 8 350.0 245 3.73 3.840 15.41 0 0 3 4 #> 19.2 8 400.0 175 3.08 3.845 17.05 0 0 3 2 #> 27.3 4 79.0 66 4.08 1.935 18.90 1 1 4 1 #> 26.0 4 120.3 91 4.43 2.140 16.70 0 1 5 2 #> 30.4 4 95.1 113 3.77 1.513 16.90 1 1 5 2 #> 15.8 8 351.0 264 4.22 3.170 14.50 0 1 5 4 #> 19.7 6 145.0 175 3.62 2.770 15.50 0 1 5 6 #> 15.0 8 301.0 335 3.54 3.570 14.60 0 1 5 8 #> 21.4 4 121.0 109 4.11 2.780 18.60 1 1 4 2


This is a subapp defined using submodules functionality in Marionette.js.
This demonstration ends here, with boilerplate covering the basic concept of working with submodules in Marionette.js app. To move forward, explore the Marionette.js reference application BBCloneMail (having its source available on Github).


Next: Basic step accessors, Previous: Stepping through the valuator, Up: Value methods [Contents][Index]
MARPA_STEP_RULE is the step type for for a rule node. The application should perform its equivalent of rule execution.
marpa_v_arg_0(v)
to
marpa_v_arg_n(v)
.
marpa_v_rule(v)
.
marpa_v_result(v)
.
Typically, the result of this step is determined by
executing the semantics for its rule on
its child values.
marpa_v_result(v)
is guaranteed to
be equal to
marpa_v_arg_0(v)
.
MARPA_STEP_TOKEN is the step type for a token node. The application’s equivalent of the evaluation of the semantics of a non-null token should be performed.
marpa_v_token(v)
.
marpa_r_alternative()
method.
See marpa_r_alternative().
Libmarpa’s “token value” will be in
stack location marpa_v_token_value(v)
.
marpa_v_result(v)
.
Often, an application will simply copy Libmarpa’s “token value”
to stack location marpa_v_result(v)
.
MARPA_STEP_NULLING_SYMBOL is the step type for a nulled node. The application’s equivalent of the evaluation of the semantics of a nulled token should be performed.
marpa_v_symbol(v)
.
marpa_v_result(v)
.
Often, an application will assign a fixed value to each
nullable symbol,
and will simply copy this fixed value to
stack location marpa_v_result(v)
.
The use of the word “nulling” in
the step type name MARPA_STEP_NULLING_SYMBOL
is problematic.
While the node must be zero-length (nulled),
the node’s symbol need not be nulling:
it may be a proper nullable.
The name of the step type would more correctly
be “MARPA_STEP_NULLED_SYMBOL”,
but it is left as is for backward compatibility.
See Nulling versus nulled.
When this is the step type, the valuator has gone through all of its steps
and is now inactive.
The value of the parse tree will be in stack location 0.
Because of optimizations,
it is possible for valuator to immediately
became inactive — MARPA_STEP_INACTIVE
could
be both the first and last step.
Once a valuator becomes inactive, it stays inactive.
The valuator is new and has yet to go through any steps.
These step types are reserved for internal purposes.
Next: Basic step accessors, Previous: Stepping through the valuator, Up: Value methods [Contents][Index]


IYCS is a computational symposium held by high schoolers from StartOnAI. IYCS provides an opportunity for students to present their AI research and projects, which are judged by experienced students, and industry leaders in AI.
Use the forms below to sign up!
Co-Organizer
Co-Organizer
Head of Technology
More information regarding Zoom links, coming soon!
Send an email if you have any questions about IYCS?


Marie uses her engineering skill set to rapidly solve problems in the medical industry. Detail-oriented while working on high-level strategy.
I lead over 20 simultaneous projects with clients ranging from small biotech firms to Fortune 500 companies. I keep projects moving through the rigorous review systems that pharma partners require and ensure we provide results with exceptional value. My dynamic role involves a great mix of responsibilities including strategic account planning, relationship management, analysis, communication, financial planning, and developing creative solutions.
Providing clients with in-depth analyses and market research insights that help outline solutions and drive executive decisions. Using my engineering background and problem-solving skills to forecast market potential, plan strategies for launch readiness, conduct market research to best answer business objectives and analyze both quantitative and qualitative data sets.
Used my engineering and problem-solving skills to forecast market potential, planned strategies for launch readiness, conducted market research to best answer business objectives and analyzed both quantitative and qualitative data sets.
Developed and launched a Resource Center, in collaboration with Biogen, on MyMSTeam (a social network for those living with MS). This Resource Center helps provide valuable information and tools to people living with MS and spread awareness about how MS can impact cognition. I was the project manager and ensured successful launch of this project by coordinating work across internal cross-departmental teams (design, product, development) as well as partnering with Biogen teammates to get the content and design finalized and approved by the Biogen franchise cPRC.
Lead this project from draft to publication on MySpondylitisTeam in partnership with UCB. Our goal is to increase awareness about spondyloarthritis and inflammatory back pain, help people recognize symptoms so that they can tell their doctors and get a diagnosis, and equip people with questions to ask their doctors. It is heartening to see so many people who may be living with spondyloarthritis find useful information and resources in this Resource Center.


Spring 2016, I worked on conjugate addition to a,B-unsaturated ketones using boronic acids and a transition metal catalyst.
Fall 2015, I worked on hydroacylation of alkynyl substrates and conjugate addition to a,B-unsaturated ketones using a transition metal catalyst.


Answer some questions to find the best track for you
OK, , now for the next step. Let's see what tracks are best for you.
We see you're from Remember, they have been matched just for you!
You can check out Epicodus' website for more information HERE.
Ruby is a programming language focused on simplicity and productivity. It has an elegant syntax that is natural to read and easy to write. The Ruby programming language is most commonly used with the Ruby on Rails framework. Ruby and Rails are typically used by younger companies and startups, especially for building interactive web applications. Companies using Ruby and Rails who have hired Epicodus graduates include New Relic, LivingSocial, and SpendWell.
Rails is a web development framework written in the Ruby language and typically referred to as “Ruby on Rails”. Rails was the first mainstream web framework to embrace the concept of convention over configuration, which improves productivity by setting sensible defaults that work for most projects. Companies using Ruby on Rails who have hired Epicodus graduates include New Relic, LivingSocial, and SpendWell.
CSS is the language for describing the presentation of Web pages, including colors, layout, fonts, and other stylistic choices. It allows one to adapt the presentation to different types of devices, such as large screens, small screens, and printers. HTML — used for building the structure of the page — and CSS — used for the visual layout — are the two basic languages involved in the building of web pages and web applications.
Web design is a process of conceptualizing, planning, and building a collection of electronic files that determine the layout, colors, text styles, structure, graphics, images, and use of interactive features that deliver pages to your site visitors. Web Design is a crucial component in web development, and students who learn design will be able to engineer positive and interactive user experiences through innovative interfaces.
Java is arguably the most popular programming language today. It's used for everything from web applications to backend services to mobile applications to embedded systems. Students who learn Java programming will be able to create dynamic web applications, and be able continue into our Android track. Companies using Java who have hired Epicodus graduates include Nike and Thetus.
Android is a mobile operating system developed by Google, based on Linux and designed primarily for touchscreen mobile devices such as smartphones and tablets. Android is the most popular operating system and used for a variety of purposes, including running mobile apps, taking photos, sending messages, etc. Companies using Android who have hired Epicodus graduates include Uncorked Studios and Rigado.
The C# ("C sharp") programming language is most commonly used with the .NET framework. Students who are well-versed in C# programming can use it to create client applications, database applications, distributed components, and much more. Students who learn C# programming and combine it with their knowledge of the .NET framework can target larger enterprises, government agencies, and companies serving enterprise and government clients for employment. Companies using C# and .NET who have hired Epicodus graduates include Windsor and Incomm.
.NET is a software framework developed by Microsoft and primarily used on Microsoft Windows. The .NET framework is used to create desktop, mobile, and web applications that run on various devices. It compiles and executes programs written in the C# and Visual Basic languages. C# and .NET are typically used by large enterprises, government agencies, and companies serving enterprises and government clients. Companies that work with C# and the .NET framework that have hired Epicodus graduates include Windsor and Incomm.
CSS is the language for describing the presentation of Web pages, including colors, layout, fonts, and other stylistic choices. It allows one to adapt the presentation to different types of devices, such as large screens, small screens, and printers. HTML — used for building the structure of the page — and CSS — used for the visual layout — are the two basic languages involved in the building of web pages and web applications.
React Description Pending...
Sorry, , but we're having trouble finding a match for you. Try reloading, selecting a different location, or changing some of your previous answers to "yes".


How To
Join the Authentication Server
after the authentication started
Enter the Token you received on the Website
You're done!
Just wait to be redirected


31+ Dodge Hemi Logo Pictures. Search results for dodge hemi logo vectors. However, dodge is the brand which has changed the logo in the course of the years many times meaning and history the first dodge logo was developed in 1910 and lasted the longest period of time.
Explore the full dodge lineup, inventory, incentives, dealership information & more. 6.2 v8 hemi® supercharged бензин 710 л.с. Search results for dodge hemi logo vectors.
Mopar heavy duty gen3 hemi lifters, oe replacement or mds delete kit, qty 16 (5.7l, 6.1l, 6.2l, 6.4l hemi engines).
You can also ask any questions about hemi engines or any of the jeep, dodge, or chrysler line up. Автомобили dodge dodge durango 2021dodge durango 2021 srt hellcat®. You can download in.ai,.eps,.cdr,.svg,.png formats. 20,789 likes · 6 talking about this.


Mount Ōyama
- Elevation: 1,252 m
- Prominence: 491 m
Source: Wikipedia
Almost everytime I hike a mountain that is more than a thousand meter high, it rains. Hiking while being cold and wet, as bad-ass as it sounds, isn’t very comfortable to do everytime. Lucky for me, this time we had the perfect weather. The sky was so clear that we expected to see Mt. Fuji.
It takes us about 1,440 Yen to go from Ookayama Station to the starting point of Mount Oyama (大山). Our route:
- Ookayama Station: Tokyu Oimachi Line to Mizonokuchi Station.
- Mizonokuchi Station: Walk to Musashimizonokuchi Station then Nambu Line to Noborito Station.
- Noborito Station: Odakyu Line to Isehara Station.
- Isehara Station: Take the bus to Oyaba Cable.
Mount Oyama is one of the most famous hiking location to Japanese people living in Tokyo, especially for families. There is two trails to the top of the mountain: Men’s trail and Woman’s trail. The Men’s trail is shorter and steeper, while the Women’s trail is longer and easier for the leg.


PERLREBACKSLASH
Section: Perl Programmers Reference Guide (1)Updated: 2009-02-12
Index Return to Main Contents
NAME
perlrebackslash - Perl Regular Expression Backslash Sequences and EscapesDESCRIPTION
The top level documentation about Perl regular expressions is found in perlre.This document describes all backslash and escape sequences. After explaining the role of the backslash, it lists all the sequences that have a special meaning in Perl regular expressions (in alphabetical order), then describes each of them.
Most sequences are described in detail in different documents; the primary purpose of this document is to have a quick reference guide describing all backslash and escape sequences.
The backslash
In a regular expression, the backslash can perform one of two tasks: it either takes away the special meaning of the character following it (for instance, "\|" matches a vertical bar, it's not an alternation), or it is the start of a backslash or escape sequence.The rules determining what it is are quite simple: if the character following the backslash is a punctuation (non-word) character (that is, anything that is not a letter, digit or underscore), then the backslash just takes away the special meaning (if any) of the character following it.
If the character following the backslash is a letter or a digit, then the sequence may be special; if so, it's listed below. A few letters have not been used yet, and escaping them with a backslash is safe for now, but a future version of Perl may assign a special meaning to it. However, if you have warnings turned on, Perl will issue a warning if you use such a sequence. [1].
It is however guaranteed that backslash or escape sequences never have a punctuation character following the backslash, not now, and not in a future version of Perl 5. So it is safe to put a backslash in front of a non-word character.
Note that the backslash itself is special; if you want to match a backslash, you have to escape the backslash with a backslash: "/\\/" matches a single backslash.
- [1]
- There is one exception. If you use an alphanumerical character as the delimiter of your pattern (which you probably shouldn't do for readability reasons), you will have to escape the delimiter if you want to match it. Perl won't warn then. See also ``Gory details of parsing quoted constructs'' in perlop.
All the sequences and escapes
\000 Octal escape sequence. \1 Absolute backreference. \a Alarm or bell. \A Beginning of string. \b Word/non-word boundary. (Backspace in a char class). \B Not a word/non-word boundary. \cX Control-X (X can be any ASCII character). \C Single octet, even under UTF-8. \d Character class for digits. \D Character class for non-digits. \e Escape character. \E Turn off \Q, \L and \U processing. \f Form feed. \g{}, \g1 Named, absolute or relative backreference. \G Pos assertion. \h Character class for horizontal white space. \H Character class for non horizontal white space. \k{}, \k<>, \k'' Named backreference. \K Keep the stuff left of \K. \l Lowercase next character. \L Lowercase till \E. \n (Logical) newline character. \N{} Named (Unicode) character. \p{}, \pP Character with a Unicode property. \P{}, \PP Character without a Unicode property. \Q Quotemeta till \E. \r Return character. \R Generic new line. \s Character class for white space. \S Character class for non white space. \t Tab character. \u Titlecase next character. \U Uppercase till \E. \v Character class for vertical white space. \V Character class for non vertical white space. \w Character class for word characters. \W Character class for non-word characters. \x{}, \x00 Hexadecimal escape sequence. \X Extended Unicode "combining character sequence". \z End of string. \Z End of string.
Character Escapes
Fixed charactersA handful of characters have a dedicated character escape. The following table shows them, along with their code points (in decimal and hex), their ASCII name, the control escape (see below) and a short description.
Seq. Code Point ASCII Cntr Description. Dec Hex \a 7 07 BEL \cG alarm or bell \b 8 08 BS \cH backspace [1] \e 27 1B ESC \c[ escape character \f 12 0C FF \cL form feed \n 10 0A LF \cJ line feed [2] \r 13 0D CR \cM carriage return \t 9 09 TAB \cI tab
- [1]
- "\b" is only the backspace character inside a character class. Outside a character class, "\b" is a word/non-word boundary.
- [2]
- "\n" matches a logical newline. Perl will convert between "\n" and your OSses native newline character when reading from or writing to text files.
$str =~ /\t/; # Matches if $str contains a (horizontal) tab.
"\c" is used to denote a control character; the character following "\c" is the name of the control character. For instance, "/\cM/" matches the character control-M (a carriage return, code point 13). The case of the character following "\c" doesn't matter: "\cM" and "\cm" match the same character.
Mnemonic: control character.
$str =~ /\cK/; # Matches if $str contains a vertical tab (control-K).
All Unicode characters have a Unicode name, and characters in various scripts have names as well. It is even possible to give your own names to characters. You can use a character by name by using the "\N{}" construct; the name of the character goes between the curly braces. You do have to "use charnames" to load the names of the characters, otherwise Perl will complain you use a name it doesn't know about. For more details, see charnames.
Mnemonic: Named character.
use charnames ':full'; # Loads the Unicode names. $str =~ /\N{THAI CHARACTER SO SO}/; # Matches the Thai SO SO character use charnames 'Cyrillic'; # Loads Cyrillic names. $str =~ /\N{ZHE}\N{KA}/; # Match "ZHE" followed by "KA".
Octal escapes consist of a backslash followed by two or three octal digits matching the code point of the character you want to use. This allows for 512 characters ("\00" up to "\777") that can be expressed this way. Enough in pre-Unicode days, but most Unicode characters cannot be escaped this way.
Note that a character that is expressed as an octal escape is considered as a character without special meaning by the regex engine, and will match ``as is''.
$str = "Perl"; $str =~ /\120/; # Match, "\120" is "P". $str =~ /\120+/; # Match, "\120" is "P", it is repeated at least once. $str =~ /P\053/; # No match, "\053" is "+" and taken literally.
Octal escapes potentially clash with backreferences. They both consist of a backslash followed by numbers. So Perl has to use heuristics to determine whether it is a backreference or an octal escape. Perl uses the following rules:
- 1.
- If the backslash is followed by a single digit, it's a backreference.
- 2.
- If the first digit following the backslash is a 0, it's an octal escape.
- 3.
-
If the number following the backslash is N (decimal), and Perl already has
seen N capture groups, Perl will consider this to be a backreference.
Otherwise, it will consider it to be an octal escape. Note that if N > 999,
Perl only takes the first three digits for the octal escape; the rest is
matched as is.
my $pat = "(" x 999; $pat .= "a"; $pat .= ")" x 999; /^($pat)\1000$/; # Matches 'aa'; there are 1000 capture groups. /^$pat\1000$/; # Matches 'a@0'; there are 999 capture groups # and \1000 is seen as \100 (a '@') and a '0'.
Hexadecimal escapes start with "\x" and are then either followed by two digit hexadecimal number, or a hexadecimal number of arbitrary length surrounded by curly braces. The hexadecimal number is the code point of the character you want to express.
Note that a character that is expressed as a hexadecimal escape is considered as a character without special meaning by the regex engine, and will match ``as is''.
Mnemonic: hexadecimal.
$str = "Perl"; $str =~ /\x50/; # Match, "\x50" is "P". $str =~ /\x50+/; # Match, "\x50" is "P", it is repeated at least once. $str =~ /P\x2B/; # No match, "\x2B" is "+" and taken literally. /\x{2603}\x{2602}/ # Snowman with an umbrella. # The Unicode character 2603 is a snowman, # the Unicode character 2602 is an umbrella. /\x{263B}/ # Black smiling face. /\x{263b}/ # Same, the hex digits A - F are case insensitive.
Modifiers
A number of backslash sequences have to do with changing the character, or characters following them. "\l" will lowercase the character following it, while "\u" will uppercase (or, more accurately, titlecase) the character following it. (They perform similar functionality as the functions "lcfirst" and "ucfirst").To uppercase or lowercase several characters, one might want to use "\L" or "\U", which will lowercase/uppercase all characters following them, until either the end of the pattern, or the next occurrence of "\E", whatever comes first. They perform similar functionality as the functions "lc" and "uc" do.
"\Q" is used to escape all characters following, up to the next "\E" or the end of the pattern. "\Q" adds a backslash to any character that isn't a letter, digit or underscore. This will ensure that any character between "\Q" and "\E" is matched literally, and will not be interpreted by the regexp engine.
Mnemonic: Lowercase, Uppercase, Quotemeta, End.
$sid = "sid"; $greg = "GrEg"; $miranda = "(Miranda)"; $str =~ /\u$sid/; # Matches 'Sid' $str =~ /\L$greg/; # Matches 'greg' $str =~ /\Q$miranda\E/; # Matches '(Miranda)', as if the pattern # had been written as /\(Miranda\)/
Character classes
Perl regular expressions have a large range of character classes. Some of the character classes are written as a backslash sequence. We will briefly discuss those here; full details of character classes can be found in perlrecharclass."\w" is a character class that matches any word character (letters, digits, underscore). "\d" is a character class that matches any digit, while the character class "\s" matches any white space character. New in perl 5.10.0 are the classes "\h" and "\v" which match horizontal and vertical white space characters.
The uppercase variants ("\W", "\D", "\S", "\H", and "\V") are character classes that match any character that isn't a word character, digit, white space, horizontal white space or vertical white space.
Mnemonics: word, digit, space, horizontal, vertical.
"\pP" (where "P" is a single letter) and "\p{Property}" are used to match a character that matches the given Unicode property; properties include things like ``letter'', or ``thai character''. Capitalizing the sequence to "\PP" and "\P{Property}" make the sequence match a character that doesn't match the given Unicode property. For more details, see ``Backslashed sequences'' in perlrecharclass and ``Unicode Character Properties'' in perlunicode.
Referencing
If capturing parenthesis are used in a regular expression, we can refer to the part of the source string that was matched, and match exactly the same thing. There are three ways of referring to such backreference: absolutely, relatively, and by name.A backslash sequence that starts with a backslash and is followed by a number is an absolute reference (but be aware of the caveat mentioned above). If the number is N, it refers to the Nth set of parenthesis - whatever has been matched by that set of parenthesis has to be matched by the "\N" as well.
/(\w+) \1/; # Finds a duplicated word, (e.g. "cat cat"). /(.)(.)\2\1/; # Match a four letter palindrome (e.g. "ABBA").
New in perl 5.10.0 is a different way of referring to capture buffers: "\g". "\g" takes a number as argument, with the number in curly braces (the braces are optional). If the number (N) does not have a sign, it's a reference to the Nth capture group (so "\g{2}" is equivalent to "\2" - except that "\g" always refers to a capture group and will never be seen as an octal escape). If the number is negative, the reference is relative, referring to the Nth group before the "\g{-N}".
The big advantage of "\g{-N}" is that it makes it much easier to write patterns with references that can be interpolated in larger patterns, even if the larger pattern also contains capture groups.
Mnemonic: group.
/(A) # Buffer 1 ( # Buffer 2 (B) # Buffer 3 \g{-1} # Refers to buffer 3 (B) \g{-3} # Refers to buffer 1 (A) ) /x; # Matches "ABBA". my $qr = qr /(.)(.)\g{-2}\g{-1}/; # Matches 'abab', 'cdcd', etc. /$qr$qr/ # Matches 'ababcdcd'.
Also new in perl 5.10.0 is the use of named capture buffers, which can be referred to by name. This is done with "\g{name}", which is a backreference to the capture buffer with the name name.
To be compatible with .Net regular expressions, "\g{name}" may also be written as "\k{name}", "\k<name>" or "\k'name'".
Note that "\g{}" has the potential to be ambiguous, as it could be a named reference, or an absolute or relative reference (if its argument is numeric). However, names are not allowed to start with digits, nor are allowed to contain a hyphen, so there is no ambiguity.
/(?<word>\w+) \g{word}/ # Finds duplicated word, (e.g. "cat cat") /(?<word>\w+) \k{word}/ # Same. /(?<word>\w+) \k<word>/ # Same. /(?<letter1>.)(?<letter2>.)\g{letter2}\g{letter1}/ # Match a four letter palindrome (e.g. "ABBA")
Assertions
Assertions are conditions that have to be true --- they don't actually match parts of the substring. There are six assertions that are written as backslash sequences.- \A
- "\A" only matches at the beginning of the string. If the "/m" modifier isn't used, then "/\A/" is equivalent with "/^/". However, if the "/m" modifier is used, then "/^/" matches internal newlines, but the meaning of "/\A/" isn't changed by the "/m" modifier. "\A" matches at the beginning of the string regardless whether the "/m" modifier is used.
- \z, \Z
-
"\z" and "\Z" match at the end of the string. If the "/m" modifier isn't
used, then "/\Z/" is equivalent with "/$/", that is, it matches at the
end of the string, or before the newline at the end of the string. If the
"/m" modifier is used, then "/$/" matches at internal newlines, but the
meaning of "/\Z/" isn't changed by the "/m" modifier. "\Z" matches at
the end of the string (or just before a trailing newline) regardless whether
the "/m" modifier is used.
"\z" is just like "\Z", except that it will not match before a trailing newline. "\z" will only match at the end of the string - regardless of the modifiers used, and not before a newline.
- \G
-
"\G" is usually only used in combination with the "/g" modifier. If the
"/g" modifier is used (and the match is done in scalar context), Perl will
remember where in the source string the last match ended, and the next time,
it will start the match from where it ended the previous time.
"\G" matches the point where the previous match ended, or the beginning of the string if there was no previous match.
Mnemonic: Global.
- \b, \B
-
"\b" matches at any place between a word and a non-word character; "\B"
matches at any place between characters where "\b" doesn't match. "\b"
and "\B" assume there's a non-word character before the beginning and after
the end of the source string; so "\b" will match at the beginning (or end)
of the source string if the source string begins (or ends) with a word
character. Otherwise, "\B" will match.
Mnemonic: boundary.
"cat" =~ /\Acat/; # Match. "cat" =~ /cat\Z/; # Match. "cat\n" =~ /cat\Z/; # Match. "cat\n" =~ /cat\z/; # No match. "cat" =~ /\bcat\b/; # Matches. "cats" =~ /\bcat\b/; # No match. "cat" =~ /\bcat\B/; # No match. "cats" =~ /\bcat\B/; # Match. while ("cat dog" =~ /(\w+)/g) { print $1; # Prints 'catdog' } while ("cat dog" =~ /\G(\w+)/g) { print $1; # Prints 'cat' }
Misc
Here we document the backslash sequences that don't fall in one of the categories above. They are:- \C
-
"\C" always matches a single octet, even if the source string is encoded
in UTF-8 format, and the character to be matched is a multi-octet character.
"\C" was introduced in perl 5.6.
Mnemonic: oCtet.
- \K
-
This is new in perl 5.10.0. Anything that is matched left of "\K" is
not included in $& - and will not be replaced if the pattern is
used in a substitution. This will allow you to write "s/PAT1 \K PAT2/REPL/x"
instead of "s/(PAT1) PAT2/${1}REPL/x" or "s/(?<=PAT1) PAT2/REPL/x".
Mnemonic: Keep.
- \R
-
"\R" matches a generic newline, that is, anything that is considered
a newline by Unicode. This includes all characters matched by "\v"
(vertical white space), and the multi character sequence "\x0D\x0A"
(carriage return followed by a line feed, aka the network newline, or
the newline used in Windows text files). "\R" is equivalent with
"(?>\x0D\x0A)|\v)". Since "\R" can match a more than one character,
it cannot be put inside a bracketed character class; "/[\R]/" is an error.
"\R" was introduced in perl 5.10.0.
Mnemonic: none really. "\R" was picked because PCRE already uses "\R", and more importantly because Unicode recommends such a regular expression metacharacter, and suggests "\R" as the notation.
- \X
-
This matches an extended Unicode combining character sequence, and
is equivalent to "(?>\PM\pM*)". "\PM" matches any character that is
not considered a Unicode mark character, while "\pM" matches any character
that is considered a Unicode mark character; so "\X" matches any non
mark character followed by zero or more mark characters. Mark characters
include (but are not restricted to) combining characters and
vowel signs.
"\X" matches quite well what normal (non-Unicode-programmer) usage would consider a single character: for example a base character (the "\PM" above), for example a letter, followed by zero or more diacritics, which are combining characters (the "\pM*" above).
Mnemonic: eXtended Unicode character.
"\x{256}" =~ /^\C\C$/; # Match as chr (256) takes 2 octets in UTF-8. $str =~ s/foo\Kbar/baz/g; # Change any 'bar' following a 'foo' to 'baz'. $str =~ s/(.)\K\1//g; # Delete duplicated characters. "\n" =~ /^\R$/; # Match, \n is a generic newline. "\r" =~ /^\R$/; # Match, \r is a generic newline. "\r\n" =~ /^\R$/; # Match, \r\n is a generic newline. "P\x{0307}" =~ /^\X$/ # \X matches a P with a dot above.
Index
This document was created by man2html, using the manual pages.
Time: 05:29:08 GMT, December 24, 2015


the and TPRTG with that elephant from al., for were are genes gene African cells copy number expression Asian (Figure was data using body evolution have which used TP/TPRTG size shown ,collab: The each not than cancer found {background TPRTG, genome protein DNA fibroblasts luciferase signaling species between enhanced large response through vector these MDM interaction proteins reads but margin this also cell either generated read transfected {border-style RTE_LA increased Figure after elephants expansion mapped px;} sequence single #ffffff;} Unmapped likely reporter segmental treated tumor into other PCR These Yes elephant, identified including may model one pcDNA.()/myc-His transcripts two ENSLAFG control evolved more risk sequencing sizes treatment DOI:http://dx.doi.org/./eLife.. animals apoptosis expanded lineage mice Data activity damage family previously resistance such suggesting then transcribed uM, Analysis Elephant N/A Proboscidean RE/Hygro] across alignment all because canonical containing data, element elements had however, inferred mechanisms putative reduced siRNA standardized total within (A) (B) A). Columbian DNA-damage RNA-Seq SRX, This dimerization doxorubicin, endogenous estimate hyrax indicate lower major mastodon maximum mouse occurred products retrogenes several some structure substitution suggest transgenic tree Consistent MDM. Renilla copies determine did duplications estimated example, least mitomycin nutlin-a, phylogenetic retrotransposition sequences studies they three time transiently transposable American For Our Proboscideans Relative aligned culture depth developing directly during empty expected extinct fibroblast following function functional human hydrophobic hyrax, increase lifespan loci mammalian mammoth mapping methods our pGL.[lucP/p predicted published px; region repeated repertoire sensitivity show thus transcript transcription woolly {font-family After BLAT Bowtie Performed Proboscideans. TP. UV-C Western al. analyses armadillo been cDNA complex contained dataset dermal difference divergence domain experiments. fold followed font-size genome. high history interpretation location many medium negative nucleotide observation pathway primers required residues results sites specific suppressor vector, via whereas ApoTox-Glo B). Illumina Luc. Myc-tagged Previous UV-C. While assayed associated blue both cancer. coincident collab: combined compared derived design, dosage double; duplication essential evolutionary exon gene. generally genes, genome, genotoxic growth identical indicating induction lack long mammals million mole normal nuclear only paired paradox prior promoter protect pseudogene rapidly rather regions role suggests supplement test their very well whole —figure (Luc.) (accession ,Publicly Archive Bayesian Broad Cellular Conception Drafting Finally, Gene HEK- Maley, Mdm NCBI Peto’s Read Research Royal Short TPRTG-specific Thus Trp University Wilcoxon additional age among antibodies available before blind border-color border-width cells. closely constraint damage. death does efficiently encode encodes estimates event expressed extra formation four gene, genes. genomes genomes. genomic has identify immunoprecipitated incubated initial interface life long-lived mean±SD. mechanism mediated models molecular no: onto organisms played primary rate regulate regulation removed resistant revising samples scaffold_ selection solid; start target tissue uniquely unsampled when while will {font-style {font-weight & (C) (Loxodonta (Procavia (Promega) (version -well ..) Abegglen Acids Acquisition Advanced Biology C). Chicago Cufflinks DNase HiSeq Identification MDM, MYA MYA), Next Nottingham Rock Similarly Society Sulak T-L TP, TPRTG. Thus, Triplex Zhang, aardvark, activation aging allow analysis ancient article, assay augmented blocks blotting bold buffer can cannot cloned collinear confidence confirm confluency consensus consisting contigs controls. correlated courier; developmental different duplications. effects elevated em; encoding escape experiments exposed five formed further grown homologous http://dx.doi.org/./University independent indicates induced interactions isoforms leading length lifespans lineage. lineages local male males manatee media min model, mutations nearly normalized p>.. pGL.[lucp/p physical potential primer pseudogenes pt;} query rat rates reads. related relatively resolution retrogenes, retrogenes. retrotransposon rounds run scrambled serial set shotgun should similar small species, stability stem-lineage stress stress, supplemented suppression template transcribed. unable under unlikely use °C/% ΔΔG α-MDM α-helix


M. Night Shyamalan Does Not Want People To Spoil ‘The Sixth Sense’ 20 Years Later
by Jamie SamhanM. Night Shyamalan stopped by “The Tonight Show With Jimmy Fallon” on Friday night where he spoke about his new TV show for Apple+ “Servant” as well as one of his more iconic films–“The Sixth Sense”.
“It is tough to watch something with me, I’ll be honest, the family doesn’t like to watch anything with me,” he revealed while talking about how his family gets annoyed. “Even if I don’t say anything, I am judging the performance and the camera angles.”
“I just get up and walk by disappointed or I will walk by the den and they are watching TV and I will yell ‘You are making a choice.'”

“Now there is a whole new generation watching, it is all new fans watching so you guys can’t say anything about it,” he told the audience.
Fallon wasn’t sure what he was and wasn’t allowed to say but Shyamalan gave him permission to say “I see dead people” but not to say what happens at the end of the movie.
Although the famous director downplayed how iconic the “dead people” line is. “It is overwritten. He is 10 but it is really something a 6 or 7-year-old would say,” he criticized himself.
The two also spoke about how when filming in Philadelphia, one of the members of The Roots accidentally messed up a scene when he exited his own house.


About
- Clodomir Santana Jr. (1)
- Federico Botta (1)
- Hugo Barbosa (1)
- Filippo Privitera (2)
- Ronaldo Menezes (1)
- Riccardo Di Clemente (1)
(1) Department of Computer Science, University of Exeter, United Kingdom
(2) Cuebiq Inc.
Contact us
If you have any questions, comments or would like to know more about our analysis, you can contact us at the following email addresses:
Riccardo Di Clemente, Exeter University
Brennan Lake, Cuebiq Inc.
Acknowledgments
This preliminary analysis is a collaboration between the Department of Computer Science of Universtiy of Exeter and Cuebiq Inc. In response to the COVID-19 crisis, Cuebiq is providing insights to academic and humanitarian groups through a multi-stakeholder data collaborative for timely and ethical analysis of aggregate human mobility patterns. We thank Pietro Gravino for the usefull discussions.


SimpSOM¶
Submodules¶
Module contents¶
SimpSOM (Simple Self-Organizing Maps) v1.3.4 F. Comitani @2017 F. Comitani @2018 F. Comitani @2019
A lightweight python library for Kohonen Self-Organising Maps (SOM).
-
SimpSOM.
run_colorsExample
(path='./')[source]¶ Example of usage of SimpSOM: a number of vectors of length three (corresponding to the RGB values of a color) are used to briefly train a small network. Different example graphs are then printed from the trained network.
-
class
SimpSOM.
somNet
(netHeight, netWidth, data, loadFile=None, PCI=0, PBC=0, n_jobs=-1)[source]¶ Bases:
object
Kohonen SOM Network class.
-
PBC
= None¶ Activate light parallelization.
-
PCI
= None¶ Switch to activate periodic boundary conditions.
-
cluster
(array, type='qthresh', cutoff=5, quant=0.2, percent=0.02, numcl=8, savefile=True, filetype='dat', show=False, printout=True, path='./')[source]¶ - Clusters the data in a given array according to the SOM trained map.
- The clusters can also be plotted.
- Args:
array (np.array): An array containing datapoints to be clustered. type (str, optional): The type of clustering to be applied, so far only quality threshold (qthresh)
algorithm is directly implemented, other algorithms require sklearn.- cutoff (float, optional): Cutoff for the quality threshold algorithm. This also doubles as
- maximum distance of two points to be considered in the same cluster with DBSCAN.
percent (float, optional): The percentile that defines the reference distance in density peak clustering (dpeak). numcl (int, optional): The number of clusters for K-Means clustering quant (float, optional): Quantile used to calculate the bandwidth of the mean shift algorithm. savefile (bool, optional): Choose to save the resulting clusters in a text file. filetype (string, optional): Format of the file where the clusters will be saved (csv or dat) show (bool, optional): Choose to display the plot. printout (bool, optional): Choose to save the plot to a file.
- Returns:
- (list of int): A nested list containing the clusters with indexes of the input array points.
-
colorEx
= None¶ Switch to activate periodic PCA weights initialisation.
-
data
= None¶ Load the weights from file, generate them randomly or from PCA.
-
diff_graph
(show=False, printout=True, returns=False, path='./')[source]¶ Plot a 2D map with nodes and weights difference among neighbouring nodes.
- Args:
- show (bool, optional): Choose to display the plot. printout (bool, optional): Choose to save the plot to a file. returns (bool, optional): Choose to return the difference value.
- Returns:
- (list): difference value for each node.
-
find_bmu
(vec)[source]¶ Find the best matching unit (BMU) for a given vector.
- Args:
- vec (np.array): The vector to match.
- Returns:
- bmu (somNode): The best matching unit node.
-
nodes_graph
(colnum=0, show=False, printout=True, path='./', colname=None)[source]¶ Plot a 2D map with hexagonal nodes and weights values
- Args:
- colnum (int): The index of the weight that will be shown as colormap. show (bool, optional): Choose to display the plot. printout (bool, optional): Choose to save the plot to a file. colname (str, optional): Name of the column to be shown on the map.
-
project
(array, colnum=-1, labels=[], show=False, printout=True, path='./', colname=None)[source]¶ - Project the datapoints of a given array to the 2D space of the
- SOM by calculating the bmus. If requested plot a 2D map with as implemented in nodes_graph and adds circles to the bmu of each datapoint in a given array.
- Args:
array (np.array): An array containing datapoints to be mapped. colnum (int): The index of the weight that will be shown as colormap.
If not chosen, the difference map will be used instead.show (bool, optional): Choose to display the plot. printout (bool, optional): Choose to save the plot to a file. colname (str, optional): Name of the column to be shown on the map.
- Returns:
- (list): bmu x,y position for each input array datapoint.
-
save
(fileName='somNet_trained', path='./')[source]¶ Saves the network dimensions, the pbc and nodes weights to a file.
- Args:
- fileName (str, optional): Name of file where the data will be saved.
-
-
class
SimpSOM.
somNode
(x, y, numWeights, netHeight, netWidth, PBC, minVal=[], maxVal=[], pcaVec=[], weiArray=[])[source]¶ Bases:
object
Single Kohonen SOM Node class.
-
get_distance
(vec)[source]¶ Calculate the distance between the weights vector of the node and a given vector.
- Args:
- vec (np.array): The vector from which the distance is calculated.
- Returns:
- (float): The distance between the two weight vectors.
-
SimpSOM.densityPeak module¶
Density Peak Clustering
A Rodriguez, A Laio, Clustering by fast search and find of density peaks SCIENCE, 1492, vol 322 (2014)
- Comitani @2017
-
class
SimpSOM.densityPeak.
collection
(coorArray, typeFunc='gaussian', percent=0.02, PBC=False, netHeight=0, netWidth=0)[source]¶ Bases:
object
Class for a collection of point objects.
-
cluster_assign
()[source]¶ Assign a cluster to each point according according to its nearest neighbour with higher density.
-
decision_graph
(show=False, printout=True)[source]¶ Calculate the decision graph, delta vs rho for the points belonging to the collection and find the cluster centers.
- Args:
- show (bool, optional): Choose to display the plot. printout (bool, optional): Choose to save the plot to a file.
-
get_clusterList
()[source]¶ Returns the indeces of the clustered points as a list.
- Returns:
- clusters (list, int): a list of lists containing the points indices belonging to each cluster
-
refd
= None¶ Make sure rhos are set before setting deltas
-
-
SimpSOM.densityPeak.
densityPeak
(sample, show=False, printout=False, percent=0.02, PBC=False, netHeight=0, netWidth=0)[source]¶ Run the complete clustering algorithm in one go and returns the clustered indeces as a list.
- Args:
- sample (array): The input dataset show (bool, optional): Choose to display the decision graph. printout (bool, optional): Choose to save the decision graph to a file.
- Returns:
- clusters (list, int): a list of lists containing the points indices belonging to each cluster
-
SimpSOM.densityPeak.
dist
(p1, p2, metric='euclid', PBC=False, netHeight=0, netWidth=0)[source]¶ Calculate the distance between two point objects in a N dimensional space according to a given metric.
- Args:
- p1 (point): First point object for the distance. p2 (point): Second point object for the distance. metric (string): Metric to use. For now only euclidean distance is implemented. PBC (bool, optional): Activate/deactivate Periodic Boundary Conditions. netHeight (int, optional): Number of nodes along the first dimension, required for PBC. netWidth (int, optional): Numer of nodes along the second dimension, required for PBC.
- Returns:
- (float): The distance between the two points.
-
SimpSOM.densityPeak.
gaussian
(p1, p2, sigma, PBC=False, netHeight=0, netWidth=0)[source]¶ Gaussian function of the distance between two points scaled with sigma.
- Args:
- p1 (point): First point object for the distance. p2 (point): Second point object for the distance. sigma (float): The scaling factor for the distance. PBC (bool, optional): Activate/deactivate Periodic Boundary Conditions. netHeight (int, optional): Number of nodes along the first dimension, required for PBC. netWidth (int, optional): Numer of nodes along the second dimension, required for PBC.
- Returns:
- (float): value of the gaussian function.
-
class
SimpSOM.densityPeak.
pt
(coordinates)[source]¶ Bases:
object
Class for the points to cluster.
-
set_delta
(coll)[source]¶ Calculate the distance of the point from higher density points and set the nearest neighbour. [Deprecated]
- Args:
- coll (collection): collection containing all the points of the dataset used to calculate the distance.
-
-
SimpSOM.densityPeak.
sigmoid
(p1, p2, sigma, PBC=False, netHeight=0, netWidth=0)[source]¶ Logistic function of the distance between two points scaled with sigma.
- Args:
- p1 (point): First point object for the distance. p2 (point): Second point object for the distance. sigma (float): The scaling factor for the distance. PBC (bool, optional): Activate/deactivate Periodic Boundary Conditions. netHeight (int, optional): Number of nodes along the first dimension, required for PBC. netWidth (int, optional): Numer of nodes along the second dimension, required for PBC.
- Returns:
- (float): value of the logistic function.
-
SimpSOM.densityPeak.
step
(p1, p2, cutoff, PBC=False, netHeight=0, netWidth=0)[source]¶ Step function activated when the distance of two points is less than the cutoff.
- Args:
- p1 (point): First point object for the distance. p2 (point): Second point object for the distance. cutoff (float): The cutoff to define the proximity of the points. PBC (bool, optional): Activate/deactivate Periodic Boundary Conditions. netHeight (int, optional): Number of nodes along the first dimension, required for PBC. netWidth (int, optional): Numer of nodes along the second dimension, required for PBC.
- Returns:
- (int): 1 if the points are closer than the cutoff, 0 otherwise.
SimpSOM.hexagons module¶
Hexagonal tiling library
- Comitani @2017
-
SimpSOM.hexagons.
coorToHex
(x, y)[source]¶ Convert Cartesian coordinates to hexagonal tiling coordinates.
- Args:
- x (float): position along the x-axis of Cartesian coordinates. y (float): position along the y-axis of Cartesian coordinates.
- Returns:
- array: a 2d array containing the coordinates in the new space.
-
SimpSOM.hexagons.
plot_hex
(fig, centers, weights)[source]¶ Plot an hexagonal grid based on the nodes positions and color the tiles according to their weights.
- Args:
fig (matplotlib figure object): the figure on which the hexagonal grid will be plotted. centers (list, float): array containing couples of coordinates for each cell
to be plotted in the Hexagonal tiling space.- weights (list, float): array contaning informations on the weigths of each cell,
- to be plotted as colors.
- Returns:
- ax (matplotlib axis object): the axis on which the hexagonal grid has been plotted.
SimpSOM.qualityThreshold module¶
Quality Threshold Clustering
L. J. Heyer, S. Kruglyak and S. Yooseph, Exploring Expression Data: Identification and Analysis of Coex-pressed Genes Genome Research, Vol. 9, No. 11, 1999, pp. 1106-1115.
- Comitani @2017
-
SimpSOM.qualityThreshold.
qualityThreshold
(sample, cutoff=5, PBC=False, netHeight=0, netWidth=0)[source]¶ Run the complete clustering algorithm in one go and returns the clustered indeces as a list.
- Args:
- sample (array): The input dataset cutoff (float, optional): The clustering cutoff distance. PBC (bool, optional): Activate/deactivate Periodic Boundary Conditions. netHeight (int, optional): Number of nodes along the first dimension, required for PBC. netWidth (int, optional): Numer of nodes along the second dimension, required for PBC.
- Returns:
- clusters (list, int): a list of lists containing the points indices belonging to each cluster


sha = "Unknown"
package_name = "kornia"
cwd = os.path.dirname(os.path.abspath(__file__))
try:
sha = subprocess.check_output(["git", "rev-parse", "HEAD"], cwd=cwd).decode("ascii").strip()
except Exception:
// the oldest one is the requirement. The versions should be inequalities.
// Once a pytorch version (in the future) breaks a kornia version, we could just
// add a maximal version.
kornia_pt_dependencies = {
"0.3.2": ">=1.5.0,<1.6.0", // torch==1.6.0 broke test/geometry/transform/test_crop.py
"0.3.1": ">=1.5.0",
"0.2.2": ">=1.4.0",
"0.1.4": ">=1.2.0",
}
// version can be overiden eg with KORNIA_BUILD_VERSION so we map each possible kornia version to the dictionary keys
def dep_version(version):


Upload a file to bulk-register devices
Simple registration
Creates all registration requests at once, then each one needs to go through regular acceptance process.Full registration
Creates all device credentials and devices using provided list of property values. Devices can start communicating with the platform immediately.


{{ categoryName}}
Architect
Architects design new buildings and the spaces around them, and work on the restoration and conservation of existing buildings.
Carpenter
Joiner, bench joiner
Carpenters and joiners make and install wooden structures, fittings and furniture.
Electrician
Electrical fitter, electrical engineer, electrical technician, installation electrician
Electricians fit, service and fix electrical equipment, circuits, machinery and wiring.
Plumber
Plumbers fit and service hot and cold water systems, heating systems and drainage networks.
Quantity surveyor
Quantity surveyors oversee construction projects, managing risks and controlling costs.


| categories:development
Using Phoenix with docker, Part 3: Containerization
This is the final part of a three part series: Part 1 - Part 2 - Part 3
Note: I am using the latest docker-compose
and the latest docker-engine
later on in this post.
Contents
- Composition
- Kitteh Dockerfile
- Database connections
- Tearing it all down
- Looking back
- One more thing
- Conclusion
So far, so good. We now have a working application that we want to containerize. At the moment, the application is feature complete, but nothing interesting related to docker
happened yet.
Composition
Before actually starting with the Dockerfile of our own codebase, let’s introduce a docker compose configuration, which is a YAML file called docker-compose.yml
:
# the database image
db:
image: "postgres:9.4"
ports:
- "5432:5432"
environment:
POSTGRES_USER: kitteh
POSTGRES_PASSWORD: kitteh
volumes_from:
- dbdata
# for persistance of database data
dbdata:
image: "postgres:9.4"
volumes:
- /var/lib/postgres
command: "true"
# the main image, this is the application
web:
build: "."
environment:
MIX_ENV: "prod"
ports:
- "4000:4000"
links:
- "db:db"
This is the naive route to take, we provide a container for the database, using the official postgres image, then use the same image for a db layer to achieve actual persistance by mounting the volumes from it into the first database image.
The application image by itself will not be buildable at the moment since it has no Dockerfile
yet, so running
docker-compose up
should pull the database related image, but then crash as the build commend fails. Or it fails as the port cannot be bound to, since you may have a local PostgreSQL instance running:
Starting kitteh_dbdata_1...
Recreating kitteh_db_1...
Building web...
Cannot locate specified Dockerfile: Dockerfile
No matter what, we should think about our applications Dockerfile
.
Kitteh Dockerfile
Remmembering the section about resizing in part 2 earlier, we know that we have ImageMagick as a dependency besides the actual Erlang VM and OTP to provide.
What do we have to ensure that our whole system works? Here is a checklist:
- all dependencies must be available as/in containers somehow
- database must be migratable and migrated to a point where it is usable
- ideally, assets are built and served via a separate process
Looking at the checklist, here are the dependencies we have:
- Erlang and OTP
- Elixir,
hex
andmix
- ImageMagick
- PostgreSQL
- nginx
For the moment, docker image sizes should not be our concern. My personal preference would be to have the most minimal footprint possible when it comes to the images, but we should focus on getting things working now:
FROM ubuntu:14.04.3
# [...]
Should be simple enough. Yes, we are using the Ubuntu image. I know it’s huge (~200MB) and it will become even larger with dependencies - but that should not stop us from using it locally.
Installing the Elixir, OTP and ImageMagick dependencies should be easy, thanks to apt-get
:
FROM ubuntu:14.04.3
# Set the locale, otherwise elixir will complain later on
RUN locale-gen en_US.UTF-8
ENV LANG en_US.UTF-8
ENV LANGUAGE en_US:en
ENV LC_ALL en_US.UTF-8
# wget for convenience, use curl if you want to
RUN apt-get -y -q install wget
# add erlang otp
RUN wget https://packages.erlang-solutions.com/erlang-solutions_1.0_all.deb
RUN dpkg -i erlang-solutions_1.0_all.deb
RUN apt-get update
RUN apt-get install -y -q imagemagick esl-erlang elixir
ADD . /app
WORKDIR /app
RUN mix local.hex --force
RUN mix deps.get --only-prod
RUN mix compile
CMD ["mix", "phoenix.server"]
Okay, not straight forward, but at least simple - it’s quite a lot of stuff to pack into the image. In the end it will be around ~500MB in size. It’s not too bad, considering a lot of layers can be reused later on, but it’s quite much, especially if you want to upload it to a hub.
Well, running it in combination with docker-compose
should now yield results:
docker-compose up
Starting d239c7b138_kitteh_dbdata_1...
kitteh_db_1 is up-to-date
Recreating kitteh_web_1...
Attaching to d239c7b138_kitteh_dbdata_1, kitteh_db_1, kitteh_web_1
db_1 | LOG: database system was shut down at 2016-02-13 18:34:32 UTC
db_1 | LOG: MultiXact member wraparound protections are now enabled
db_1 | LOG: database system is ready to accept connections
db_1 | LOG: autovacuum launcher started
db_1 | LOG: received smart shutdown request
db_1 | LOG: autovacuum launcher shutting down
db_1 | LOG: shutting down
db_1 | LOG: database system is shut down
db_1 | LOG: database system was shut down at 2016-02-13 18:43:57 UTC
db_1 | LOG: MultiXact member wraparound protections are now enabled
db_1 | LOG: autovacuum launcher started
db_1 | LOG: database system is ready to accept connections
db_1 | LOG: received smart shutdown request
db_1 | LOG: autovacuum launcher shutting down
db_1 | LOG: shutting down
db_1 | LOG: database system is shut down
db_1 | LOG: database system was shut down at 2016-02-13 18:50:18 UTC
db_1 | LOG: MultiXact member wraparound protections are now enabled
db_1 | LOG: autovacuum launcher started
db_1 | LOG: database system is ready to accept connections
web_1 | [info] Running Kitteh.Endpoint with Cowboy on http://localhost:4000
web_1 | [error] Could not start watcher because "node" is not available. Your Phoenix application is still running, however assets...
We forgot assets. We’ll deal with it down the road. For now, visisting your localhost should yield the original web application we created earlier - at least it’s first page.
Uploading shouldn’t work - there is no migrated database yet and you should get a connection error when trying to upload a picture.
If you are to lazy to create your own Dockerfile
, please see the tag 06-dockerizing-kitties
here.
Database connections
We should see if we can make the database work.
The database images used at the moment utilize the official postgres image. The docker-compose.yml
file will set environment variables that should correspond to the values in your local prod.secret.exs
file.
Tip: Read environment variables into that configuration like this:
# ./config/prod.secret.exs
# [...]
# Configure your database
config :kitteh, Kitteh.Repo,
adapter: Ecto.Adapters.Postgres,
hostname: "db", # see the links section in docker-compose.yml for the name
username: System.get_env("DB_USER")
password: System.get_env("DB_PASSWORD"),
database: "kitteh_prod",
pool_size: 20
For our use case this can stay hard coded as this file is not checked into git anyway. It might be different if one would choose to utilize a CI.
However, making sure that the database is reachable is key, since we need to migrate it (the database image actually already contains a database named after the environment variable set here - come to think of it, it should probably be the same as in the configuration)
Note: I encountered some problems when trying to use the $POSTGRES_DB
variable. for some reason it always fell back to the $POSTGRES_USER
. I have not yet figured out why. These problems only occured while using the version 1 of the compose configuration.
Long story short, if we run
# use -d for detached mode
docker-compose up
we can attach to the running docker container using exec
like this:
# container name may vary
docker exec -it kitteh_web_1 /bin/bash
or
docker-compose run web /bin/bash
and then run
# add MIX_ENV=prod if you did not change the docker-compos.yml entry
mix ecto.migrate
inside the container to finally migrate the database. If this fails, try to make sure that $MIX_ENV
is set to prod
in the environment.
If we now try to upload a picture again, it should work. You can of course always run migrations using
docker-compose run web mix ecto.migrate
directly.
Tearing it all down
If we restart our composition, we should take note that our database data should have been persisted. But if we reload the image url we just generated, the image will be gone, as the actual images are not persisted in the container.
Since we ignored assets completely so far, it’s now time to give this a more serious thought.
First, we add an nginx
entry to our docker-compose.yml
(I chose the official image):
# [...]
nginx:
image: "nginx:1.9.11"
# [...]
# now what?
Container structure
Let’s talk about the basic idea first:
We’ll use a data volume container that can be used by both the not-yet-existing nginx
container, as well as the existing web
container. The web
container will write to the data volume container, the nginx
container will read the data and serve it appropiately. With this, we can individually backup the data container if we wanted to and the data is persisted.
Here is the checklist:
- Create a data container with volumes for both
web
andnginx
containers. - Mount volumes into both containers
- Create an
nginx
container that has a site config for serving the contents of the volume mounted
Creating the volume container
Before we add our container, we should reevaluate where exactly we want to store our images.
Inside the container, the current uploads
folder resides within _build/prod/lib/kitteh/priv/static/uploads
. That’s quite a handful to type.
So let’s change that first. Luckily, we only have to change the target_path
function in the page_controller.ex
:
# ./web/controllers/page_controller.ex
# [...]
defp target_path do
if Mix.env == :prod do
"/var/uploads/"
else
Application.app_dir(:kitteh, "priv") <> "/static/uploads/"
end
end
We’ll create another container in our docker-compose.yml
that is built from scratch:
# [...]
assets:
image: "busybox"
volumes:
- /var/uploads
command: "true"
If everything went well, the images should be preserved between container restarts.
We can now create an nginx docker image. We’re going to use a custom Dockerfile to do so - we can store it alongside the codebase.
Docker 1.10
This is where things get interesting - Docker was recently updated to v1.10.0 - and along with it came an update to docker-compose
with a new version of the syntax, allowing for custom dockerfile directives in the build
object of the configuration.
I decided to give it a spin. At this point you’ll have to upgrade as well, otherwise you cannot use the version 2 of the configuration syntax.
Note: If you do not want to upgrade, you can always build yourself a local image using the Dockerfile.nginx
and update the version 1 docker-compose.yml
accordingly.
The updated configuration can be found in this commit.
Configuring nginx
I went with a super simple nginx configuration file, which is bogus, but should work nonetheless.
Running the containers
A simple
docker-compose up
should bring up our creation. I also tagged this point of development for your convenience if you do not wish to code along.
Assets and images
Wait, what about assets? We still get something along the lines of
[error] Could not find static manifest at "/app/_build/prod/lib/kitteh/priv/static/manifest.json". Run "mix phoenix.digest" after building your static files or remove the configuration from "config/prod.exs."
in the logs.
And what about actually serving the images? With out current setup, the nginx
will use the mounted volume under its own /var/uploads
to serve the content of this volume. Running
docker-compose run nginx ls -l /var/uploads
should yield a list of images. The nginx
container itself is exposed at port 4001
- thanks to the configuration yaml.
So, assuming a file named LargeSuperbYellowTurkishAngora.jpeg
actually exists in the data container a simple
http get http://localhost:4001/LargeSuperbYellowTurkishAngora.jpeg
should give the image back to you. If you are not using httpie yet, I do recommend it!
So, for now, we can use this and modify our code, specifically the view that renders the image url (called asset_url
):
# /web/views/page_view.ex
def asset_url(image) do
"/img/" <> image.token
end
One of the drawbacks of not having an asset pipeline similar to Rails is that we cannot specify an integrated asset host (that is not completely true, as we can easily just reuse an Application
or environment variable for that). However, it’s not as simple as just changeing config.host
in your production.rb
, as the pipeline is not entangled with the framework.
But that will not stop us, even if it means hardcoding values:
# /web/views/page_view.ex
def asset_url(image)
"http://localhost:4001/" <> Path.basename(image.path)
end
In a production environment, I would probably assign a real subdomain name, like “images.kitt.eh” or something along these lines.
Rebuilding and restarting the containers should now present the very same image, but served from the nginx container.
Note: At this point I noticed a flaw in the Dockerfile
for the main application - Compilation was not done properly, so I had to reintroduce an env variable to mark production compilation.
Assets
Phoenix purposefully does not integrate an asset pipeline and instead delegates to a library called brunch
(see brunch.io) to handle static assets like JavaScript and CSS.
I personally disagree with the choice of brunch - I’d have used webpack myself, but one cannot be too picky when others provide work that is essentially free.
Anyway, we have different options for solving the assets problem domain:
- Reusing the nginx container (and having the assets land on “image.kitt.eh”)
- Introducing a second nginx container (and introducing “assets.kitt.eh” in proudction)
Serve everything via the web stack.- Serve everything via a CDN.
I am not really going to talk about option number 4. Using a CDN (like Cloudfront) is usually the way to go with bigger production setups as you just cannot match the server infrastructure that is provided to you via these services.
I am inclined to use option number 2. Assets and images should be different in my opinion and - and that is even more crucial to me - updating the assets should not affect the data within the uploads container. Depending on your requirements, option 3 should also be a very possible solution.
However, let’s introduce a third Dockerfile
, called Dockerfile.assets
(we also rename the existing Dockerfile.nginx
to Dockerfile.uploads
):
FROM nginx:1.9.11
# override the default nginx config with our own
ADD nginx.assets.conf /etc/nginx/nginx.conf
RUN mkdir /logs
EXPOSE 80
CMD ["nginx", "-g", "daemon off;"]
It’s essentially the same one as for uploads. The nginx.conf
looks similar as well, but defines proper mimetypes for all the sublocations (css & js).
I also updated the docker-compose.yml
accordingly to include the second nginx to serve assets later on.
We run into a problem here: so far we just assumend that all the assets are there, ready to go - however, we cannot run brunch build --production
, as this is an npm
dependency. Up until now, we just assumed it was there, as we had it installed anyway. However, a fresh clone of the repo cannot assume that. To keep it a bit shorter, I will omit the steps necessary to make this work, but to give you an idea, here are the options you’d have:
- Install
nodejs
in the web container (bloating it even more) and then runnpm install
in it (bloating it even more). This might be an option if you want everything self-contained and ready to go after cloning the repo - Live with it and require the user (or a CI for that matter) to have node installed on their system when building the image
I chose option 2 for this exercise and documented it in the README.
Unfortunately, I found digesting all the assets and then putting them into /var/assets
at buildtime impossible, as this requires all the containers to be up.
Thereby running
docker-compose run web mix phoenix.digest -o /var/assets
is crucical for this to work and can only be done when the containers are running.
Note: A solution to this is to invert the dependency and build the separate nginx container with the digested assets in it, mounting its volumes into the web container later on.
However, asset digestion can be done on demand this way - the volume container will just keep the last version of the digested assets.
Running the containers now should ideally provide the static assets already:
# the hash may vary
http get http://localhost:4002/js/app-<hash>.js
should yield our JavaScript. Same goes for our css.
We also need to update our prod.exs
environment to make the manifest file known:
# /config/prod.exs
config :kitteh, Kitteh.Endpoint,
http: [port: {:system, "PORT"}],
url: [host: "example.com", port: 80],
cache_static_manifest: "/var/assets/manifest.json"
The asset paths in app.html.eex
have to be updated to make this work with the nginx serving our assets (remember, this would be “assets.kitt.eh” in production):
<!-- /web/templates/layout/app.html.eex -->
<!-- [...] -->
<link rel="stylesheet" href="<%= static_url(@conn, "/css/app.css") %>">
<!-- [...] -->
<script src="<%= static_url(@conn, "/js/app.js") %>"></script>
This requires a small addition to prod.exs
(static_url
):
# /config/prod.exs
config :kitteh, Kitteh.Endpoint,
http: [port: {:system, "PORT"}],
url: [host: "example.com", port: 80],
static_url: [host: "localhost", port: 4002],
cache_static_manifest: "var/assets/manifest.json"
Rebuilding and running our containers again should now yield results - all the uploaded images are served from an nginx and all of the assets are served from another. also, everything is persisted independently.
Neat.
For your convenience, I added a tag here so that you can compare progress yourselves.
Looking back
We have done a lot so far. We coded a small uploader application and put most of the dependencies into containers.
Here is the list of containers used:
- A PostgreSQL for persistance
- An nginx for serving uploaded images
- A second nginx for serving static assets like JavaScript and CSS
- A web application container that holds the application
There are also 3 volume containers that are used to share data between the containers and provide persistance.
Cleaning up
We do have some prolems with the setup as well:
- This might be overkill for such a small application and you should consider bare metal for this setup (or Heroku if you feel so inclined)
- The web application container is huge
- Two nginx are not necessarily better than one
- There is no proper configuration management yet - most of the stuff we need is hardcoded either in our application or the docker files.
In fact, let’s clean up some problems right here, right now. Introduce a .dockerignore
file and add the following list of patterns:
# Generate on crash by the VM
erl_crash.dump
# Static artifacts
node_modules/
That reduces the web
images size by about 4MB, as the node_modules
are not needed in the image. Woah.
The problem here is that we are using a full Ubuntu image (which I chose for convenience reasons like installing via apt-get
). For procatical purposes we should probably use an alpine-based image to build our application container.
There is also SSL yet, we’re purely on relying on HTTP - which is not a production setup I would endorse in 2016.
Finally, our main attention should probably also focus on removing items from the codebase in the web container that we do not need - mainly the actual codebase, as Elixir compiles to BEAM code and we would only need that. Just for comparison, we are talking about 18MB in the case of kitteh.
It depends on how much convenience you want - ultimately, reducing tha base image size should be the first goal in my opinion.
One more thing
We could have also used Exrm to bundle and prepare a release into a container. I will (probably) explore this in a later entry.
Conclusion
If you read this far - thanks! I hope you found the material I provided interesting and you can take something away for your own projects.
If there are questions for this particular series, you can either write me an email or use the issues for the project on Github. Being not a very skillful Elixir developer, I am always thankful for Pull Requests and comments.
You have reached the end of this series. You might be interested in how everything started and how it turned out to be implemented after all.


A.I. Artificial Intelligence
Reviewed by Harvey Karten Warner Bros/Amblin-Stanley Kubrick productions Director: Steven Spielberg Writer: Ian Watson, Brian Aldiss (short story "Supertoys Last All Summer Long" Cast: Haley Joel Osment, Jude Law, Frances O'Connor, Brendan Gleeson, Jake Thomas, William Hurt, Daveigh Chase
We all know how automation--a form of robotics, if you will--can make a difference in our lives. The I.B.M. tech-help phone line tells us that our call is important to them and proves it by playing a half hour of Vivaldi's "The Four Seasons" while we wait for the attention of a mere mortal. We wheel our S.U.V. into the car wash and watch as large arms reach out to apply soap, water and wax to our vehicle and brush up our advanced transportation without the need for people to do much more than push some buttons. An invisible hand reaches out to pick out the pins in the local bowling alley which we have maliciously knocked down as we pitch strike after strike. Even the very car that we get polished up without the need for the intervention of people is put together by robots. What can we expect robotics to do in the decades that lie ahead in our new century?
In a sci-fi film which marries the detached, epic vision of Stanley Kubrick to the sentimental child-centered themes of Steven Spielberg, "A.I.: Artificial Intelligence," shows us in an awe-inspiring way how robots can be used to help us emotionally as well as physically. In a world that finds love in short supply, it was only a matter of time before some scientific genius would transcend the current rage of cloning by creating a mechanical child equipped without the strength needed to assemble a car but with the more important vitality of love. As that scientist, Professor Hobby (William Hurt) tells an assemblage of his peers at the corporation of which he is the creative director about the importance of receiving affection: "God made Adam to love Him." In the brief opening moments of this fairly long film, anyone in the audience with the slightest grasp of literature or of sentimental filmmaking will realize that the a child-robot created with the ability to give love will change not only the mother who craves this affection but the boy himself (or itself).
"A.I.: Artificial Intelligence" was the final project conceived by Stanley Kubrick to be screened some time after what became the late, great director's swan song, "Eyes Wide Shut." Kubrick had taken his good friend Steven Spielberg into his confidence to discuss a story about an advanced technology, one which would blend science (the creation of robots to do jobs that human beings would prefer not to do) and humanity (the creation of robots to furnish emotional warmth to those who choose to buy them). After Kubrick's death, Spielberg took over the project which one producer calls an attempt by him to "embrace and pay homage to Stanley." To those in the audience who are familiar with works such as "E.T." and "Jurassic Park"--the latter focusing on the grandchildren of paleontologists who tag along with the scientists to inspect an island amusement park not realizing that anything can go wrong--the picture that now emerges looks mostly like Spielberg's, but not without the insistent entrace of Kubrick's motifs.
A fifty-minute sequence takes place largely within the comfortable suburban home of a young couple, Henry Swinton (Sam Robards) and his wife Monica (Frances O'Connor). Since their only son, a fatally ill Martin (Jake Thomas) has been cryogenically frozen pending such time as a cure can be found for his disease, they long for another boy to love. Their dream comes true as Henry surprises his wife by buying an android named David (Haley Joel Osment) from the scientist Dr. Hobby (William Hurt) during a period of declining resources and limited land space causing the government to restrict the growth of human population. Like Pinocchio--which is one among many fables that are called upon to inform the movie's motifs--David decides that in return for the love he is giving, he wants love in return. The only way to get this affection is to become a real boy himself. He will ultimately devote quite a bit of time to this quest.
This initial segment in what is obviously a sci-fi tale is surprisingly the most compelling despite its sappiness, perhaps because after David is introduced to his new owners, and later to a now-recovered "brother" Martin, Mr. Spielberg appears to take a long break, handing the movie over to the Kubrick touch. The middle segment of what is really a trilogy, all centered on David, is cold, detached, unemotional, as we are introduced to two locations that are futuristic but not representing the sort of prospect that most of us would welcome. (Some spoilers follow.) After David is tearfully abandoned by Monica together with a cuddly robotic bear named Teddy (voice of Jack Angel) which steals every scene in which he appears, he is transported to a couple of ticky-tacky locations that only Mad Max could love. He meets with and ultimately saves the life of Gigolo Joe (Jude Law), a robot designed to give massages to the ladies and then some, an automaton who tells his customers that after a session with him they will never again want a real man. He does not even need a CD player to help seduce the women. He merely shakes his head to one side and out comes romantic music. When a robot-hating Luddite named Lord Johnson-Johnson (Brendan Gleeson) emcees a killing field within a stadium that could have been the arena for "Gladiator"--shooting the trapped robots out from cannons into fiery deaths (which could give the kiddies attending the movie nightmares)--David leads Joe to safety and continues his odyssey in search of the Blue Fairy, who like the Wizard of Oz is said to be able to fulfill his dream and turn him into other than he now is.
There are numerous references to fables in addition to The Wizard of Oz and Pinocchio, some self-referential, as Spielberg winks at the audience challenging us to catch them. One place is called "Strangelove's" while the mayhem in Lord Johnson- Johnson's emporium reminds us of "A Clockwork Orange." The execution of the obsolete robots calls to mind the termination of people over 30 years old in "Logan's Run," while the general concept often recalls "Close Encounters of the Third Kind."
"A.I.: Artificial Intelligence" is photographed by Janusz Kaminski for the most realistically, showing a future that aside from the robots is not so much more advanced than our own era and, in fact, given the research and results of cloning experiments is hardly beyond the grasp of audience imagination. The car which Monica uses to take David away, to dump him into the countryside like an unwanted pet but with a warning to avoid corporate employees who are instructed to destroy trashed toys, is a cool vehicle but in no way one that cannot now be designed if the powers-that-be wanted to take a leap forward by a decade of so. Even the scene of New York, now submerged because of melting ice caps, is recognizable by the Chrysler building which majestically keeps its head out of the water defying the elements, and considerable attention has been given to produce a digitalized milieu inhabited by a citizenry alternately brutalized by the spectacle of robots being malicious destroyed while some, like the scientist Dr. Hobby, remain and compassionate.
Haley Joel Osment, twelve years old at the time of the filming, has lost some of the cuteness that made him a compelling figure in "The Sixth Sense," and in fact the movie was put on the fast track to be complete in twenty weeks instead of the full year that was originally planned because this wonderful young man is quickly showing signs of physical maturity. He remains an effective center for the story, showing more depth than even the Australian actress Frances O'Connor in the role of the mother-- whose emotional depth consisted of being either teary-eyed or ambivalent.
By the time we are taken to the final scene, which brings us to a civilization two millennia ahead inhabited by figures looking as though they could be at home on Venus, we may not be blamed for wondering whether this craggy but generally imaginative and visually splendid and mature picture has a single, all-embracing motif. John Williams's music is annoyingly ubiquitous if varied but does not provide a clue to the Kubrick-Spielberg design. Nonetheless "A.I." is not a picture to ignore or pass up. This is sci-fi in the seasoned, deliberately-paced spirit of "Contact" rather than the absurdly childish productions such as the "Alien" series, featuring a varied range of set designs, musical compositions, situations, and moods.
Rated . Running time: 143 minutes. (C) 2001 by Harvey Karten<EMAIL_ADDRESS>The review above was posted to the
rec.arts.movies.reviews newsgroup (de.rec.film.kritiken for German reviews).
The Internet Movie Database accepts no responsibility for the contents of the
review and has no editorial control. Unless stated otherwise, the copyright
belongs to the author.
Please direct comments/criticisms of the review to relevant newsgroups.
Broken URLs inthe reviews are the responsibility of the author.
The formatting of the review is likely to differ from the original due
to ASCII to HTML conversion.
Related links: index of all rec.arts.movies.reviews reviews


{% unless page.comments_locked == true %}
If you have a question concerning the content of this page, please feel free to contact me.
Leave a comment


We’re hosting an in-person meetup at NIEC on 28th October, from 10 am - 3 pm, to gather all free and open source enthusiasts from the Delhi city to contribute to open source projects together.
Hacktoberfest is an annual month-long event to celebrate and encourage more developers to become involved with open source software. It’s open for anyone, especially beginners just getting started with open source!
In this meetup, we’ll take a beginner friendly session on Git and GitHub.
The most active contributors, who make at least 4 PR’s merged, will receive T-shirts sponsored by Digital Ocean.
Room No. 4202 , Northern India Engineering College
Time:10 am - 3 pm
For Partnership/Sponsorship opportunities, contact Shivam Singhal at<EMAIL_ADDRESS>, mob:-9310945854


April 14, 2019
This file extends the introduction I gave to simulating data from coin flips. Now, the goal is to show how the basics extend and can easily be used for helpful parts of study design planning.
Simulating More Complex Models
We can easily extend the basic concepts of simulating from a simple population to simulating data from a more complex data structure. This is how we can simulate a simple regression model with one outcome (Y) and five predictors (x1, x2, etc.).
library(lavaan)
# what we think the relationship are
hypothesized.model <- '
y ~ .1*x1 + .2*x2 + -.2*x3 + .5*x4
y ~~ 1*y
'
## Simulate 1 dataset
pop.data <- simulateData(hypothesized.model,
sample.nobs=1000,
seed=2 ## Seed is for reproducing results
)
## Visualize scatterplot matrix
plot(pop.data)
## Estimate the model we care about
fitted.model <- '
y ~ x1 + x2 + x3 + x4
'
fitted.out <- sem(
fitted.model,
data=pop.data,estimator="ML")
## Get summary statistics of results
summary(fitted.out, rsquare=TRUE)
Simulation for Power Calculation
library(simsem)
## Simulate 100 datasets of size 100 from the population
act.power <- sim(nRep=100,
generate=hypothesized.model,
model=fitted.model,
n =100,
lavaanfun = "sem")
## Extract summary information
sim.parms<-summaryParam(act.power,alpha = 0.05,detail=TRUE)
sim.parms
Simulation for Sample Size Determination
Now, we want to calculate the power for a range of sample sizes. By calculating power across a range we are able to identify the sample size needed to reach power of .80 fora particular parameter of interest. Let’s try to answer this question: What is sample size I need if I want to detect the effect of x2 on y?
## Simulate data for a range of sample sizes
# seq(100,500,10): pick samples sizes from 100 to 500 going by 10 each time, so we will have 100, 110, 120, 130, ... , 500.
# the rep(..., 100): each sample size is replicated 100 times, so we will effectively run 100 datasets with sample size 110, then 100 datasets with sample size 500, etc.
act.n <- sim(model=fitted.model,
generate=hypothesized.model,
n = rep(seq(100,300,10), 100),
lavaanfun = "sem")
## Create a power plot
plotPower(act.n, alpha=0.05, powerParam="y~x2")
abline(h=.8, lwd=2, lty=2)
## Rerun for more fine grained information across wider range of sample sizes
act.n <- sim(model=fitted.model,
n =seq(10,1000,10),
generate=hypothesized.model,
lavaanfun = "sem")
act.pwr.n <- getPower(act.n, alpha = 0.05)
findPower(act.pwr.n, iv="N", power=0.8)
Next is simulating data from factor models and structural equation models.


When sending a message via MQTT, the message will publish directly to MQTT from the browser to the topic chattopic/mqtt.
When sending a message via AWS, the message will be posted to an AWS Lambda via API Gateway. The Lambda will then publish the message to the topic chattopic/lambda.
MQTT
This window is subscribing to the topic chattopic/mqtt.
AWS Lambda
This window is subscribing to the topic chattopic/lambda.


On August 29, 1949, the USSR conducted their first successful nuclear test, called RDS-1 or "First Lightning." The United States called it "Joe-1." The test took place at Semipalatinsk.
Both the United States and the Soviet Union began hydrogen bomb research and development by the early 1950s.


Data control
Populate the data and run the application. When querying for the data a progress bar is rendered. As long as it animates, the main thread is safe.
Populate the data and run the application. When querying for the data a progress bar is rendered. As long as it animates, the main thread is safe.


Lightning network topology, its creation and maintenance
Transcript By: Bryan Bishop
Tags: Topology, Lightning, Routing
Category: Conference
Carla Kirk-Cohen
video: https://www.youtube.com/watch?v=j2l_Ut4k1qI
Introduction
Alright. Give me a second to test this. Alright. Antoine has taken you through the routing layer. I’m going to take you through what the lightning network looks like today. This is the current topology of the network and how this came about, and some approaches for maintaining hte network and making the graph look like we want it to look.
Lightning brief overview
There’s ideally only two transactions involved in any lightning channel, the commitment transaction and the closure transaction. We want to make sure there’s as few opens and closes on the chain as possible. We want to make sure the topology of the network enables routing so that we don’t have to constantly keep opening and closing channels to make payments. It’s really constraining when you’re on the lightning network and you have to open a new channel to make a payment, and you end up clogging the chain anyway.
We open and close channels wisely.
Public vs private channels
Public channels are known to the network, and announced on the gossip protocol using a channel announcement. This announcement is agreed by both of the nodes in the channel. They intend for people to route through it. This is very different from a private channel, which is not announced to the network. If you have a peer trying to announce a private protocol, you just don’t sign the channel info and other clients won’t route through it unless there’s both signatures.
When we look at the topology of the network, we can only look at the public topology really unless someone else volunteers private data. It’s kind of annoying because we want to see the really interesting things in the internet. It may be possible to deduce or infer information about private channels through hints.
There’s also personal vs routing nodes. A public node or a routing node would setup some public channels and manage capital well, and hope to earn a lot of fees. Fees are very low in lightning but hopefully one day we can have higher nodes and running these nodes can be incentive compatible. It would be great if people earned feeds for running lightning nodes. I think with intelligent channel creation, you can definitely earn a lot more.
Path finding and routing
We use source-based routing in the lightning network. However, you have a scarcity of funds. This means there’s going to be a lot of failures through LN because nobody knows the funds. However, there is something you can do called a probing attacks where you route payments through a specific edge and you fail it with a bad preimage but you route it through with various routes and the minute you hit a range where the payments stop going back to you or failing with a certain range, then you know that’s the amount available in the channel. At the moment, it’s possible to figure out balances even though we’re trying to hide it. It can deanonymize things if you can trace balances through the network.
We optimize for fees and timelocks. There’s a channel update message on the network. This is how we figure out how we’re going to route.
Topology matters
The current topology really matters. It affects the ease of path finding, the success of payments, and it influences fees. If you’re poorly connected to the network, then you’re probably going to be paying more fees. It also is related to the resilience of the network and it opens us up to topology analysis by surveillance companies if we’re not careful.
Topology overview
There’s 5600 nodes, about 35218 public channels, and 959 public BTC sitting in those channels. I used the tool LNTopology found on github to run an analysis and figure out what the network looks like. At least the public network.
The diameter is defined as the shortest longest path. The longest path you can take in a network between any two nodes. In lightning, we’re limited to 20 hops. We’re presently at about 11. So we’re well within the limit.
Extent is the other one which is longest shortest path, and this value in the lightning network is 6. This gives you a picture of the extent of the network.
I don’t see why someone would want to drive up the diameter of the lightning network, but just creating two nodes and creating a very large chain at the edge of the graph. It would cost you a bit to do this, but again these are worst case metrics possibly produced by nodes not necessarily actual actors in the network.
A graph of the degree distribution shows that there’s a small number of nodes with a ton of channels. This is indicative that we have a hub-and-spoke network. There’s a collection of hubs. A hub is one big node that has a bunch of connections, and then the smaller nodes connect to the hub and they have fewer connections. This is fairly supported by the assortativity of the graph, which is a value between -1 and 1. This measures the tendency of nodes to connect to nodes with similar degrees. If it’s a positive number, it means low degree nodes connect to high degree nodes ((or is it the other way around)).
Topology creation
When LN first started out, the topology was achieved through manual connections. People connected with other people they knew. Then in 2017, lnd one of the three main implementations, added something called autopilot which automates the way in which you connect to nodes. It worked based on preferential attachment where you would become more likely to connect to nodes that already have a lot of channels open and have liquidity. This drove the development of the hub-and-spoke network. At the time, it was intended for autopilot and the original commit said “we probably shouldn’t do this but here’s a first attempt at this”. It has significantly influenced the network topology.
Also another aspect of topology/channel creation is liquidity providers. There are some companies that will open a channel to you, like Biig or Bitrefill, which will accept an out-of-band payment where if you pay them they will open up a channel with you providing liquidity. Only a few people were doing this; a lot of channels got opened up and now lnbig has 20-24% of the liquidity of the lightning network in their channels.
Network maintenance
Maintenance is an interesting topic. There’s a need to distinguish between channel health and network health. If you’re running a node, you want to connect to people who are online a lot. If you’re connected to offline nodes, then it becomes pretty unreasonable. You want to route and get fees.
Network health is about setting up a topology that is conducive to routing and resists network attacks, regardless of how healthy individual channels are.
Changing topology has a high cost. There would be at least 35,000 transactions on chain if the lightning network all channels were closed. So that’s not really reasonable, and it doesn’t make sense as a way to scale bitcoin of course. I think we need to open and close channels more wisely.
Opening channels
The problem with manually opening channels is that people tend to deanonymize themselves and their node metadata. So this is straight deanonymization. People should be responsible for their own privacy, but we should provide a default for less educated users.
The other issue is that lightning is a payments graph; it’s not a social graph. There’s really interesting work on the twitter and facebook social graphs but we don’t really know what the graphs on lightning will tend to look like. You might not open a channel with your friend, but rather businesses you work with. You will probably perpetuate a hub-and-spoke network anyway because companies tend to have a power law about them.
Second degree preferential attachment is about connecting to nodes that are peers and are connected to the well-connected hubs. This will help receive better positioning in the network, but it’s slightly better than just connecting to hubs exclusively.
Graph metrics
c-lightning is looking at using graph metrics to inform open/close channel decisions. There’s two categories- connections that go to the network as a whole, and then the one as you as a channel alone. There’s one mode which is randomly connect, and another one that connects in a way to decrease the diameter of the graph. That’s interesting because you can close the diameter. But it is a bit gameable, because someone can costlessly create a large diameter for the network. I’m sure we can get around this fairly easily, like have some sanity checks in there.
Another one is “betweenness centrality” which is good for a node that wants to maximize fees. You want to connect to nodes with high betweenness centrality. They tend to sit on a high amount of the shortest paths on the network. If you connect to such a node, then you will probably sit on many of the shortest paths as well.
Also preferential attachment is still an option.
Node sorting
Another direction which takes a different approach is node scoring. There’s something called “buzz scores” which are publicly released scores, but how they do it is not public. But yo ucan make some guesses, like uptime is probably involved. Make sure your nodes are online. I think they do some basic probing to figure out who are the more balanced nodes on the network. I think they run some small probes every 2 weeks and they try to figure out who has more liquidity and they score them well because of this.
Again, this is potentially gameable, if you know what metrics they are using, then you can try to increase your score. But it’s hard to see an attack here. You get a good score by behaving well. But maybe you can temporarily increase your performance while you get probed, but not sooner.
I think the mobile app currently uses this scoring list and is publicly available.
Closing channels
I am pretty interested in closing channels.  Also, you have the opportunity cost of losing out on future routing fees. All of these are difficult things to think about as a person. I worked on a mainnet lightning integration and we had these channels that weren’t doing anything.
I worked on a system for scoring channels which will hopefully inform autopilot about making a decision when to close a channel. I looked at four dimensions.
Reliability: the uptime of the remote peer in the channel.
Success rate: HTLC success rates that the channel provides, versus the number of failures. Obviously, the failures could be further down the line, but maybe this is connected to a channel that keeps failing and then we should want to close that even though we’re not the problem.
Profit/volume: this is based on fees earned and total amount moved through the channel. This needs to be calibrated to how much BTC you have in the channel.
Utility: how much personal use we get from a channel. This isn’t particularly important for a routing node, but if you have a channel open and you make a payment every day to buy coffee or a beer or something, you don’t want to close the channel just because there’s no routing fees because your other options might cost more than running your own node.
Conclusion
The lightning network seems to have a hub and spoke topology. Topology is really hard to change. It’s expensive to change this.