PyTorch is an open source machine learning framework that is primarily used for building deep learning models. The framework is built on top of the Torch library and is implemented in Python, with support for C++ and CUDA.

The main C++ classes in PyTorch are:

1. Tensor: This is the core object in PyTorch and represents a multi-dimensional array. Tensors are the basic building blocks of a PyTorch model and are used to store and manipulate data.
2. Autograd: This is PyTorch’s automatic differentiation engine, which allows developers to compute gradients of tensors with respect to a loss function. The autograd module also provides a set of functions for computing gradients of complex functions.
3. nn.Module: This is a base class for all neural network modules in PyTorch. It provides a convenient way to define and organize layers of a neural network, as well as a set of useful methods for training and evaluating the model.
4. Optimizer: This is a class that implements various optimization algorithms, such as stochastic gradient descent (SGD), Adam, and Adagrad. The optimizer is used to update the parameters of a model during training.
5. DataLoader: This is a utility class that provides an efficient way to load and preprocess large datasets for training a model. The DataLoader class can be used to batch and shuffle data, as well as to apply various transformations to the data.

PyTorch’s autograd engine implements a variant of reverse-mode automatic differentiation, which is also known as backpropagation. This algorithm efficiently calculates the gradients of the output with respect to each input variable by traversing the computational graph in reverse order, propagating the gradients backwards through each operation using the chain rule.

Chainer Variables implements this implementation of the chain-rule of differentiation and their doc explains this well – https://docs.chainer.org/en/latest/guides/variables.html

Step by step example of automatic differentiation – https://stats.stackexchange.com/questions/224140/step-by-step-example-of-reverse-mode-automatic-differentiation

https://pytorch.org/tutorials/beginner/nlp/deep_learning_tutorial.html

Getting started with CUDA via pytorch –

>>> import torch
>>> torch.cuda
<module 'torch.cuda' from '/opt/conda/envs/pytorch/lib/python3.10/site-packages/torch/cuda/__init__.py'
>>> torch.cuda.is_available()
True
>>> torch.cuda.get_device_name()
'Tesla V100-SXM2-16GB'
>>> torch.cuda.memory_allocated()
0
>>> torch.cuda.get_device_properties(0).total_memory
16935419904
>>> import pynvml
>>> from pynvml import *
>>> nvmlInit()
>>> h = nvmlDeviceGetHandleByIndex(0)
>>> torch.cuda.mem_get_info()
(16112549888, 16935419904)
>>> var1=torch.FloatTensor([1.0,2.0,3.0]).cuda()
>>> var1
tensor([1., 2., 3.], device='cuda:0')
>>> var1.device
device(type='cuda', index=0)
>>> import torch.nn as nn