Movement of the Robotic Arm

Movement:

1. Keep a heap of dry garbage (a mix of plastic, paper and leaves) in a tray. The tray is rectangular and light grey in color. The tray is always kept at a specific position. Keep the tray in a closed environment  where the light source is from the left as well as from the right of the tray. The robotic arm is mounted such that it's left to right movement is parallel to the tray's longer edge. 

2. Start from the top of the heap. The "heap top" can be calibrated to the left-of-larger-edge of the tray and height as high as the robotic arm can go up in the air. This larger edge of the tray is closer to the robotic arm. 

3. Now start swiping the camera horizontally, parallel to the tray's larger edge which is closer to the robotic arm. While swiping pick and segregate the objects which are identified as "paper", "plastic" or "leaves". 

• The "identification" process is as follows. Keep swiping the arm and capture three photos with 50ms delay. If two of the photos give us the same predictions, let's say "paper", then classify the object as paper. 
• The "picking" process is as follows: Pause the robotic arm when you receive two equal readings out of three. Let's say the three readings are "leaves", "paper", "paper". Stop the arm. Open the gripper to half of its maximum. And bring the gripper down until it touches the object. (Assuming a touch sensor is attached to the arm). Pick the object and move it to the appropriate bin (plastic, paper or leaves). The bin coordinates will be calibrated in the beginning of the experiment. (Timeout, if the object can not be picked up in 2 attempts and move forward by lifting the robotic arm.) After picking one object, put it in the appropriate bin. Lift the robotic arm up in the air, to its maximum height and start moving to the right until it reaches the rightmost edge of the tray. Once it reaches the rightmost edge, move the robotic arm 3 cm towards the farther edge of the tray. And start traversing to the left from edge to edge. Continue capturing photos along the way. And keep the 2 out of 3 logic intact, to segregate the objects. Complete one traversal of the tray by moving 3 cm everytime from closest to farthest edge and moving from "left to right" and "right to left" of the tray. Like-wise repeat this traversal across the edges three times. Pick and segregate the objects. Count the remaining objects, the segregated objects and "time required" to check the efficiency of the robotic arm. Note the reason why the remaining objects did not get segregated? Is that due to time crunch? Or the three times traversal is less to pick all the objects? Is the gripper opening insufficient? Or due to objects overlay?

Below is how the Robotic Arm will look like, with mobile phone on the clamp and the gripper.

Goal - Integrate with Arduino uno, to capture images from Arducam Mini 2MP Plus, and save it on the computer. 

Step 1:

Wiring Connections - Arducam Mini 2MP Plus (OV2640) to Arduino Uno

VCC: Connect to 5V on Arduino Uno.

GND: Connect to Ground (GND).

CS (Pin 10): Connect to digital Pin 10 (SPI chip select).

MOSI (Pin 11): Connect to digital Pin 11.

MISO (Pin 12): Connect to digital Pin 12.

SCK (Pin 13): Connect to digital Pin 13.

SDA (A4): Connect to Analog Pin A4 (I2C data).

SCL (A5): Connect to Analog Pin A5 (I2C clock).

Step 2:

Official video streaming sketch - ArduCAM_Mini_Video_Streaming.ino. Upload it on Arduino Uno.

Step 3:

Arducam's PC host software via USB - "Arducam EVK SDK" to save the images programmatically on to the PC. "Arducam EVK SDK" trigger to capture a photo when the gripper is at a specific position , after every one second programmatically using python, C or C++. 

Step 4:

Think about the display of photos. [TBD]

Flow for Core Training and Incremental Training and Predictions using the above setup

1. Arducam EVK SDK triggers the camera to capture photos when the gripper is at a specific position after every one second. 

2. A previously created core segregation model is then used for prediction of objects "Leaves", "Paper" and "Plastic". According to prediction, a signal is sent to the robotic arm to move the object into "Leaves Bin", "Paper Bin" or "Plastic Bin".

3. When the number of objects go beyond 10, an "incremental training" process is triggered and the 

"segregation model" gets refined. And further predictions happen with this model. And the objects are segregated in a refined way. 

 

A block diagram for high level programming of "Predicting Robotic Arm With Camera" is given below.

 

 

Discussions over prototyping and sellable product:

• We need to envision what our prototype looks like. 
• Can we get a camera directly connected to the Computer, rather than the camera going to the Arduino and the Arduino going to the computer?
• Initially in the prototype phase, we will just have one layer of garbage. It will not be a heap. But the question is, the stake-holders might find it very basic. I mean, it is demonstrating only "classification", which is already demonstrated using mobile app. The real challenge in Garbage Collection is to sort out a "heap" of garbage. The option of touch sensor is good. Whenever the gripper of the robotic arm touches the garbage, the object will be picked up and sorted.
• We may pick the object up and when it comes closer to camera, we can capture a photograph and classify it. Rather than first capturing the photograph and then picking the object up. 
• out of box - can we design a filter through which only plastic can be separated or only paper can be separated. Currently we are thinking of separating the garbage along the lines of "processing done by human being". Is some different approach feasible?
• To reach the final product, it will be a three step process - 1. Have only one layer of garbage and sort it out 2. Have a heap of garbage to sort it out 3. Collect remote objects, make a heap and sort it out
• "speed" is going to be a crucial factor here. 
• We can capture a photograph of the heap and move the robotic arm wherever there is plastic in the photograph. Pick all the visible plastic in one go. Then capture another photograph and pick all the visible plastic. But the problem with this approach is, whenever an object is picked up from the heap, the surrounding objects will move. And the previous photograph will no longer be valid.
• In real life, there will be millions of objects to be sorted out in one day. So sorting the objects one by one, does not seem to be feasible. 
• How to calculate the efficiency of robotic arm "objects/hour"?
• Another approach to segregate is to divide the whole tray into a grid. Start from the peak of the heap. Which is mostly in the middle of the tray. Start classifying objects from this middle-top position, with the help of camera and touch sensor. Slowly come down classifying the objects. 
• Why can't we have mobile attached to the robotic arm? - We need to do the following processes - Capture an image, inference it, move the robotic arm as per inference. We can capture the picture and inference it on mobile. Can't we use the same mobile attached in the robotic arm, to give appropriate instructions to the robotic arm after inferencing? We will save the price of an additional GPU with this setup. And the whole setup will be compact. 
• We need to do inference on the laptop and give the final segregation instruction to the robotic arm. We can not dispatch "the robotic arm along with the laptop" in production. In future we might need to make the robotic arm working with GPU (instead of a laptop) and sell it. 
• We also discussed T shaped mounting positions of the Gripper and mobile. The camera can capture further pictures, meanwhile the gripper will pick the object and place it in an appropriate bin. But for this to happen, the part of T which carries the gripper should be flexible enough. Such that it can move without moving the camera. 

GPU + Camera + Arduino + development cost or future support cost per kit

4500 + 2878 + 340 + X