KaiserScience

Home » coding

Category Archives: coding

Advertisements

Programming Labs for Physics

coding-snippet

These labs were designed by Prof. Chris Orban for Physics 1250 at The Ohio State University at Marion. They are useful at the high school and college level. No calculus knowledge or prior programming experience is required.

The nice part about these programming labs is that there is no software to install. The compiling and executing and visualization is all done within your web browser! This is accomplished using a programming framework called p5js.org which is very similar to C/C++.

Introduction to the p5.js programming framework

Related video:

The Physics of Video Games! STEM coding.

.

Advertisements

The future of photography on phones depends on coding

Note to students: When we talk about coding, we mean computer programming (“writing code.”) But more specifically, we mean using code that uses sophisticate mathematics.

________________

From “The Future of Photography is Code”

Devin Coldewey, 10/22/2018

What’s in a camera? A lens, a shutter, a light-sensitive surface and, increasingly, a set of highly sophisticated algorithms. While the physical components are still improving bit by bit, Google, Samsung and Apple are increasingly investing in (and showcasing) improvements wrought entirely from code. Computational photography is the only real battleground now.

The reason for this shift is pretty simple: Cameras can’t get too much better than they are right now, or at least not without some rather extreme shifts in how they work. Here’s how smartphone makers hit the wall on photography, and how they were forced to jump over it.

Oppo N3 cellphone camera

The sensors in our smartphone cameras are truly amazing things. The work that’s been done by the likes of Sony, OmniVision, Samsung and others to design and fabricate tiny yet sensitive and versatile chips is really pretty mind-blowing. For a photographer who’s watched the evolution of digital photography from the early days, the level of quality these microscopic sensors deliver is nothing short of astonishing.

But there’s no Moore’s Law for those sensors. Or rather, just as Moore’s Law is now running into quantum limits at sub-10-nanometer levels, camera sensors hit physical limits much earlier. Think about light hitting the sensor as rain falling on a bunch of buckets; you can place bigger buckets, but there are fewer of them; you can put smaller ones, but they can’t catch as much each; you can make them square or stagger them or do all kinds of other tricks, but ultimately there are only so many raindrops and no amount of bucket-rearranging can change that.

Samsung Galaxy S camera sensor

Photo by Petar Milošević, from Wikimedia, commons.wikimedia.org/wiki/File:Samsung_Galaxy_S_camera_sensor.jpg

Sensors are getting better, yes, but not only is this pace too slow to keep consumers buying new phones year after year (imagine trying to sell a camera that’s 3 percent better), but phone manufacturers often use the same or similar camera stacks, so the improvements (like the recent switch to backside illumination) are shared amongst them. So no one is getting ahead on sensors alone.

From Photos on Grey scale cellphone cameras

Image from FLIR Machine Vision, https://www.ptgrey.com/white-paper/id/10912

Perhaps they could improve the lens? Not really. Lenses have arrived at a level of sophistication and perfection that is hard to improve on, especially at small scale. To say space is limited inside a smartphone’s camera stack is a major understatement — there’s hardly a square micron to spare. You might be able to improve them slightly as far as how much light passes through and how little distortion there is, but these are old problems that have been mostly optimized.

The only way to gather more light would be to increase the size of the lens, either by having it A: project outwards from the body; B: displace critical components within the body; or C: increase the thickness of the phone. Which of those options does Apple seem likely to find acceptable?

In retrospect it was inevitable that Apple (and Samsung, and Huawei, and others) would have to choose D: none of the above. If you can’t get more light, you just have to do more with the light you’ve got.

Isn’t all photography computational?

The broadest definition of computational photography includes just about any digital imaging at all. Unlike film, even the most basic digital camera requires computation to turn the light hitting the sensor into a usable image. And camera makers differ widely in the way they do this, producing different JPEG processing methods, RAW formats and color science.

For a long time there wasn’t much of interest on top of this basic layer, partly from a lack of processing power. Sure, there have been filters, and quick in-camera tweaks to improve contrast and color. But ultimately these just amount to automated dial-twiddling.

The first real computational photography features were arguably object identification and tracking for the purposes of autofocus. Face and eye tracking made it easier to capture people in complex lighting or poses, and object tracking made sports and action photography easier as the system adjusted its AF point to a target moving across the frame.

These were early examples of deriving metadata from the image and using it proactively, to improve that image or feeding forward to the next.

In DSLRs, autofocus accuracy and flexibility are marquee features, so this early use case made sense; but outside a few gimmicks, these “serious” cameras generally deployed computation in a fairly vanilla way. Faster image sensors meant faster sensor offloading and burst speeds, some extra cycles dedicated to color and detail preservation and so on. DSLRs weren’t being used for live video or augmented reality. And until fairly recently, the same was true of smartphone cameras, which were more like point and shoots than the all-purpose media tools we know them as today.

The limits of traditional imaging

Despite experimentation here and there and the occasional outlier, smartphone cameras are pretty much the same. They have to fit within a few millimeters of depth, which limits their optics to a few configurations. The size of the sensor is likewise limited — a DSLR might use an APS-C sensor 23 by 15 millimeters across, making an area of 345 mm2; the sensor in the iPhone XS, probably the largest and most advanced on the market right now, is 7 by 5.8 mm or so, for a total of 40.6 mm2.

Roughly speaking, it’s collecting an order of magnitude less light than a “normal” camera, but is expected to reconstruct a scene with roughly the same fidelity, colors and such — around the same number of megapixels, too. On its face this is sort of an impossible problem.

Improvements in the traditional sense help out — optical and electronic stabilization, for instance, make it possible to expose for longer without blurring, collecting more light. But these devices are still being asked to spin straw into gold.

Luckily, as I mentioned, everyone is pretty much in the same boat. Because of the fundamental limitations in play, there’s no way Apple or Samsung can reinvent the camera or come up with some crazy lens structure that puts them leagues ahead of the competition. They’ve all been given the same basic foundation.

All competition therefore comprises what these companies build on top of that foundation.

Image signal processing cellphone Camera math software Apple

Image as stream

The key insight in computational photography is that an image coming from a digital camera’s sensor isn’t a snapshot, the way it is generally thought of. In traditional cameras the shutter opens and closes, exposing the light-sensitive medium for a fraction of a second. That’s not what digital cameras do, or at least not what they can do.

A camera’s sensor is constantly bombarded with light; rain is constantly falling on the field of buckets, to return to our metaphor, but when you’re not taking a picture, these buckets are bottomless and no one is checking their contents. But the rain is falling nevertheless.

To capture an image the camera system picks a point at which to start counting the raindrops, measuring the light that hits the sensor. Then it picks a point to stop. For the purposes of traditional photography, this enables nearly arbitrarily short shutter speeds, which isn’t much use to tiny sensors.

Why not just always be recording? Theoretically you could, but it would drain the battery and produce a lot of heat. Fortunately, in the last few years image processing chips have gotten efficient enough that they can, when the camera app is open, keep a certain duration of that stream — limited resolution captures of the last 60 frames, for instance. Sure, it costs a little battery, but it’s worth it.

Access to the stream allows the camera to do all kinds of things. It adds context.

Context can mean a lot of things. It can be photographic elements like the lighting and distance to subject. But it can also be motion, objects, intention.

A simple example of context is what is commonly referred to as HDR, or high dynamic range imagery. This technique uses multiple images taken in a row with different exposures to more accurately capture areas of the image that might have been underexposed or overexposed in a single exposure. The context in this case is understanding which areas those are and how to intelligently combine the images together.

This can be accomplished with exposure bracketing, a very old photographic technique, but it can be accomplished instantly and without warning if the image stream is being manipulated to produce multiple exposure ranges all the time. That’s exactly what Google and Apple now do.

Something more complex is of course the “portrait mode” and artificial background blur or bokeh that is becoming more and more common. Context here is not simply the distance of a face, but an understanding of what parts of the image constitute a particular physical object, and the exact contours of that object. This can be derived from motion in the stream, from stereo separation in multiple cameras, and from machine learning models that have been trained to identify and delineate human shapes.

These techniques are only possible, first, because the requisite imagery has been captured from the stream in the first place (an advance in image sensor and RAM speed), and second, because companies developed highly efficient algorithms to perform these calculations, trained on enormous data sets and immense amounts of computation time.

What’s important about these techniques, however, is not simply that they can be done, but that one company may do them better than the other. And this quality is entirely a function of the software engineering work and artistic oversight that goes into them.

DxOMark did a comparison of some early artificial bokeh systems; the results, however, were somewhat unsatisfying. It was less a question of which looked better, and more of whether they failed or succeeded in applying the effect. Computational photography is in such early days that it is enough for the feature to simply work to impress people. Like a dog walking on its hind legs, we are amazed that it occurs at all.

But Apple has pulled ahead with what some would say is an almost absurdly over-engineered solution to the bokeh problem. It didn’t just learn how to replicate the effect — it used the computing power it has at its disposal to create virtual physical models of the optical phenomenon that produces it. It’s like the difference between animating a bouncing ball and simulating realistic gravity and elastic material physics.

Why go to such lengths? Because Apple knows what is becoming clear to others: that it is absurd to worry about the limits of computational capability at all. There are limits to how well an optical phenomenon can be replicated if you are taking shortcuts like Gaussian blurring. There are no limits to how well it can be replicated if you simulate it at the level of the photon.

Similarly the idea of combining five, 10, or 100 images into a single HDR image seems absurd, but the truth is that in photography, more information is almost always better. If the cost of these computational acrobatics is negligible and the results measurable, why shouldn’t our devices be performing these calculations? In a few years they too will seem ordinary.

If the result is a better product, the computational power and engineering ability has been deployed with success; just as Leica or Canon might spend millions to eke fractional performance improvements out of a stable optical system like a $2,000 zoom lens, Apple and others are spending money where they can create value: not in glass, but in silicon.

Double vision

One trend that may appear to conflict with the computational photography narrative I’ve described is the advent of systems comprising multiple cameras.

This technique doesn’t add more light to the sensor — that would be prohibitively complex and expensive optically, and probably wouldn’t work anyway. But if you can free up a little space lengthwise (rather than depthwise, which we found impractical) you can put a whole separate camera right by the first that captures photos extremely similar to those taken by the first.

Now, if all you want to do is re-enact Wayne’s World at an imperceptible scale (camera one, camera two… camera one, camera two…) that’s all you need. But no one actually wants to take two images simultaneously, a fraction of an inch apart.

These two cameras operate either independently (as wide-angle and zoom) or one is used to augment the other, forming a single system with multiple inputs.

The thing is that taking the data from one camera and using it to enhance the data from another is — you guessed it — extremely computationally intensive. It’s like the HDR problem of multiple exposures, except far more complex as the images aren’t taken with the same lens and sensor. It can be optimized, but that doesn’t make it easy.

So although adding a second camera is indeed a way to improve the imaging system by physical means, the possibility only exists because of the state of computational photography. And it is the quality of that computational imagery that results in a better photograph — or doesn’t. The Light camera with its 16 sensors and lenses is an example of an ambitious effort that simply didn’t produce better images, though it was using established computational photography techniques to harvest and winnow an even larger collection of images.

Light and code

The future of photography is computational, not optical. This is a massive shift in paradigm and one that every company that makes or uses cameras is currently grappling with. There will be repercussions in traditional cameras like SLRs (rapidly giving way to mirrorless systems), in phones, in embedded devices and everywhere that light is captured and turned into images.

Sometimes this means that the cameras we hear about will be much the same as last year’s, as far as megapixel counts, ISO ranges, f-numbers and so on. That’s okay. With some exceptions these have gotten as good as we can reasonably expect them to be: Glass isn’t getting any clearer, and our vision isn’t getting any more acute. The way light moves through our devices and eyeballs isn’t likely to change much.

What those devices do with that light, however, is changing at an incredible rate. This will produce features that sound ridiculous, or pseudoscience babble on stage, or drained batteries. That’s okay, too. Just as we have experimented with other parts of the camera for the last century and brought them to varying levels of perfection, we have moved onto a new, non-physical “part” which nonetheless has a very important effect on the quality and even possibility of the images we take.

_______________

Related articles

Your Smartphone Should Suck. Here’s Why It Doesn’t. (Wired magazine article)

Great images! How can we use slow shutter speed technique during day time?

Great images!! The Exposure Triangle – A Beginner’s Guide.

_______________

This website is educational. Materials within it are being used in accord with the Fair Use doctrine, as defined by United States law.
§107. Limitations on Exclusive Rights: Fair Use. Notwithstanding the provisions of section 106, the fair use of a copyrighted work, including such use by reproduction in copies or phone records or by any other means specified by that section, for purposes such as criticism, comment, news reporting, teaching (including multiple copies for classroom use), scholarship, or research, is not an infringement of copyright. In determining whether the use made of a work in any particular case is a fair use, the factors to be considered shall include: the purpose and character of the use, including whether such use is of a commercial nature or is for nonprofit educational purposes; the nature of the copyrighted work; the amount and substantiality of the portion used in relation to the copyrighted work as a whole; and the effect of the use upon the potential market for or value of the copyrighted work. (added pub. l 94-553, Title I, 101, Oct 19, 1976, 90 Stat 2546)

 

Emergent phenomenon

Thomas T. Thomas writes:

From our perspective at the human scale, a tabletop is a flat plane

Tabletop

,but at the atomic level, the flat surface disappears into a lumpy swarm of molecules.

Tunneling Microscope Atoms Silicon

Aficionados of fractal imagery will understand this perfectly: any natural feature like the slope of a hill or shore of a coast can be broken down into smaller and smaller curves and angles, endlessly subject to refinement. In fractal geometry, which is driven by simple equations, the large curves mirror the small curves ad infinitum.

The emergent property is not an illusion…  The flatness of the tabletop is just as real—and more useful for setting out silverware and plates—than the churning atoms that actually compose it. The hill and its slope are just as real—and more useful for climbing—than the myriad tiny angles and curves, the surfaces of the grains of sand and bits of rock, that underlie the slope.

Emergent property works on greater scales, too. From space the Earth presents as a nearly perfect sphere, a blue-white marble decorated with flashes of green and brown, but still quite smooth. That spherical shape only becomes apparent from a great distance. Viewed from the surface, it’s easy enough for the eye to see a flat plane bounded by the horizon and to focus on hills and valleys as objects of great stature which, from a distance of millions of miles, do not even register as wrinkles.

Emergent properties come into play only when the action of thousands, millions, or billions of separate and distinct elements are perceived and treated as a single entity. “Forest” is an emergent property of thousands of individual trees. The concept of emergent properties can be extremely useful to describe some of the situations and events that we wrestle with daily.

thomastthomas.com/Emergent_Properties

also

http://www.pbs.org/wgbh/nova/nature/emergence.html

Examples

Conway’s game of life

https://en.wikipedia.org/wiki/Conway%27s_Game_of_Life

http://emergentuniverse.wikia.com/wiki/Conway%27s_Game_of_Life

http://www.scholarpedia.org/article/Game_of_Life

http://www.conwaylife.com/

 

BOIDS: Birds flocking

Boids Background and Update by Craig Reynolds

http://www.red3d.com/cwr/behave.html

http://www.emergentmind.com/boids

Coding: 3 Simple Rules of Flocking Behaviors: Alignment, Cohesion, and Separation

https://en.wikipedia.org/wiki/Flocking_(behavior)

Classical physics

Classical physics is an emergent property of quantum mechanics

Emergent properties

 

External links

Online Interactive Science Museum about Emergence

 

Learning Standards

2016 Massachusetts Science and Technology/Engineering Curriculum Framework

Appendix VIII Value of Crosscutting Concepts and Nature of Science in Curricula

In grades 9–12, students can observe patterns in systems at different scales and cite patterns as empirical evidence for causality in supporting their explanations of phenomena. They recognize that classifications or explanations used at one scale may not be useful or need revision using a different scale, thus requiring improved investigations and experiments. They use mathematical representations to identify certain patterns and analyze patterns of performance in order to re-engineer and improve a designed system.

Next Gen Science Standards HS-PS2 Motion and Stability

Crosscutting Concepts: Different patterns may be observed at each of the scales at which a system is studied and can provide evidence for causality in explanations of phenomena. (HS-PS2-4)

A Framework for K-12 Science Education

Scale, proportion, and quantity. In considering phenomena, it is critical to recognize what is relevant at different measures of size, time, and energy and to recognize how changes in scale, proportion, or quantity affect a system’s structure or performance…. The understanding of relative magnitude is only a starting point. As noted in Benchmarks for Science Literacy, “The large idea is that the way in which things work may change with scale. Different aspects of nature change at different rates with changes in scale, and so the relationships among them change, too.” Appropriate understanding of scale relationships is critical as well to engineering—no structure could be conceived, much less constructed, without the engineer’s precise sense of scale.

Dimension 2, Crosscutting Concepts, A Framework for K-12 Science Education: Practices, Crosscutting Concepts, and Core Ideas (2012)

 

http://necsi.edu/guide/concepts/emergence.html

 

Boolean logic

How to program in Scratch, using Boolean logic

  • Boolean operators include:
  • AND, OR, NOT,  < ,  = ,  >
  •    In other words, is one sprite touching some other thing? The answer by definition must be true or false.
  •    In other words, is one sprite touching something of a certain color? The answer by definition must be true or false.
  •  In other words, is a certain key being pressed?
  •  In other words, is the mouse being used?
  • Why use Boolean operators?
  •   To focus a search, particularly when your topic contains multiple search terms.
  •   To connect various pieces of information to find exactly what you’re looking for.
  • https://sites.google.com/a/onalaskaschools.com/tech/boolean-search-tools
  • .
  • https://ircutp.wordpress.com/utp-irc-faqs/boolean-operators/

 

A Boolean block is a hexagonal block (shaped after the Boolean elements in flowcharts)
The block contains a condition. The answer to the condition will be either true or false.
.
It’s important to determine if a statement (expression) is “true” or “false”.
Ways to determine TRUE and FALSE are prevalent in all kinds of decision making.
A mathematically precise way of asking if something is TRUE or FALSE is called a Boolean operation.
It is named after George Boole, who first defined an algebraic system of logic in the mid 19th century.
.
Boolean data is associated with conditional statements. For example, the following statement is really
a set of questions that can be answered as TRUE or FALSE.
IF (I want to go to a movie) AND (I have more than $10) THEN (I can go to the movie)
.
We can combine several “boolean” statements that have true/false meaning into a single statement
using words like AND and OR, and NOT).
“If I want to go to the movie AND I have enough money, then I will go to the movie.”
BOTH conditions have to evaluate to true (have to be true) before the entire expression is true.
.
Some terms you already learned in math are really Boolean operators
Less than < [ ] < [ ] > Equal to < [ ] = [ ] > Greater than < [ ] < [ ] >
For example: (The height of a building) < 20 meters
For any building we look at, this statement will either be true or false.

Go through what each Boolean block does (page 68)

 

External links

Boolean: Lesson 13: GetCoding

Books

Book “Adventures in Coding”, Eva Holland and Chris Minnick, Wiley, 2016. Pages 50-59

Learning Standards

Massachusetts

Computational Thinking 6-8.CT.c.2 Describe how computers store, manipulate, and transfer data types and files (e.g., integers, real numbers, Boolean Operators) in a binary system.

CSTA K-12 Computer Science Standards

CT.L2-14 Examine connections between elements of mathematics and computer science
including binary numbers, logic, sets and functions.

CPP.L2-05 Implement problem solutions using a programming language, including: looping behavior, conditional statements, logic, expressions, variables, and functions.

 

AI assignment

On a separate sheet of paper please answer these questions (or fill in the blanks)

AI – readings

1. AI is intelligence exhibited by machines. It doesn’t mean that ____________ .

2. What does it mean for a computer to be intelligent?

3. People can do more than solve problems: we are aware, sentient, sapient, and conscious. What does it mean for someone to be sapient?

4. What does it mean for someone to be sentient?

5. Some AI problems have successfully been solved. People are so used to them that we often don’t even call them AI.  You may already be using some AIs in your own life. List & clearly describe two examples of these AIs.

6. There is a field of computer science called “philosophy of artificial intelligence.” List 2 of the questions that field is working to answer.

7. How are strong and weak AIs different?

8. In our resource, look at the 2 articles we link to “How to Help Self-Driving Cars Make Ethical Decisions”, or “What Will It Take to Build a Virtuous AI?” Summarize the first article, in your own words, in 3 well-written paragraphs. Summarize the second article for homework.

 

Coding weather forcecasting

Coding weather forcecasting

GR2Analyst Tornado GRLEVELX

Intro

articles

Unleash Your Inner Geek With These Excellent Weather Radar Programs

http://thevane.gawker.com/unleash-your-inner-geek-with-these-excellent-weather-ra-1628544982

Sample programs

Gibson Ridge Software, LLC (GRS) was created in March 2005 and produces viewers for weather radar data. GRS applications include GRLevel2 for viewing Level II radar data and GRLevel3 for viewing Level III data. Both viewers feature high speed, high quality radar displays with an intuitive user interface. All GRS applications are written in multithreaded C++ using the base Windows APIs for speed and efficiency.

GRLevelX

ProductsGRLevel3 2.00

GR2Analyst 2.00

GREarth

GRLevel2

 

View storm features in 3D with GR2Analyst’s Volume Explorer

Display 124 nm and 248 nm sweeps for a complete view of hurricanes and other large-scale phenomena

Smoothing reconstructs features distorted by standard bin-oriented displays 

See severe thunderstorm and tornado warnings along with details of Nexrad-identified threats

Related articles

Radar

 

Learning standards

Massachusetts Earth Science

8.MS-ESS2-5. Interpret basic weather data to identify patterns in air mass interactions and the relationship of those patterns to local weather.

8.MS-ESS2-6. Describe how interactions involving the ocean affect weather and climate on a regional scale, including the influence of the ocean temperature as mediated by
energy input from the Sun and energy loss due to evaporation or redistribution via
ocean currents.

Common Core Math Skills

STANDARD (CCSS.MATH.PRACTICE) INTRODUCTION TO PROGRAMMING THE EV3

MP1 Make sense of problems and persevere in solving them Chapters are all based around solving real-world robot problems; students must make sense of the problems to inform their solutions

MP2 Reason abstractly and quantitatively Programming requires students to reason about physical quantities in the world to plan a solution, then calculate or estimate them for the robot

MP4 Model with mathematics Many processes, including the process of programming itself, must be systematically modeled on both explicit and implicit levels

NGSS

HS-ETS1-2. Design a solution to a complex real-world problem by breaking it down into smaller, more manageable problems that can be solved through engineering.

 

Code.Org lessons

Hit the ground running: Coding lessons from Code.Org

We’re using Blockly, a visual coding language.

code-org-puzzle-coding-example

 

What are “conditionals”?  “On One Condition” If-then-else conditional flowchart lesson

Course 2. Stage 3: Angry Birds coding the path through the maze.

Course 2, Stage 6: Maze Loops

Course 2, Stage 7. Artist loops

Course 2, Stage 8. Bee Loops.

Course 2, Stage 10. Bee Debugging.

Course 2, Stage 11. Artist debugging.

Course 2, Stage 13. Bee Conditionals.

Course 2, Stage 16. Flappy Birds

Code.org Stage 19 Artist – nested loops

Code.Org: Course 4 Intro to programming

Course 4: Stage 6, Artist variables

Course 4, Stage 7 : Play Lab variables

Course 4, Stage 9. Bee “For” loops, and counters

Course 4, Stage 10. Artist. “For” Loops.

Course 4. Stage 11. Play Lab “For” loops

Course 4. Stage 12. Artist Functions
Homework: Write a paragraph explaining how loops work, how WHILE loops work,
and how DO…WHILE loops work.

Tutorialspoint: Loops and Chapter 8: Loops. Introduction to Programming

Course 4. Stage 14. Artist with parameters

Course 4. Stage 16. Bees – Functions with parameters
Homework: What is a “Hello, World!” program?

Excelwithbusiness.com: Say “Hello, world!”.
How would  we tell a computer to write “Hello, World!” in Blockly?

PCAdvisor How to code-with-google-blockly

Course 4, Stage 19: variables super challenge
Homework :Go to TutorialsPoint (link below) Choose “programming environment”.
(1) What’s the purpose of a text editor?
(2) What’s the purpose of a compiler?
(3) What’s the purpose of  an interpreter?
Tutorialspoint.com lessons

Course 4. Stage 20. For Loops Super Challenge: Can you get Skrat to his acorn using as few blocks as possible?

Course 4. Stage 21. Super Challenge – Functions and Parameters

Course 4. Stage 2. EXTREME CHALLENGE. No rules. No clues. Just the challenge!

Hour of code programs

tba

Resources

Code.Org: Course 2 Intro to programming

Accelerated Intro to Computer Science: Code.Org